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ABSTRACT » 

An annotated bibliography is organized to match the 
format of the Standards for Evaluati^s of Educational Programs., 
Pro-iects, and IJaterials, including sections on utility, feasibility, 
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administrators, evaluators, curriculum specialists, school board 
members, legislators, counselors, leaders of educational ^ 
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INTRODUCTION 



As a means of improving educational evaluations, 12 professional organizations 
in 197-5 appointed a 17 member joint committee and charged it with devising 
standards for educational evaluations. Following five years of development- 
including extensive review, field tests and hearings—the Standards for ^ 
Evalu ations of Educational Programs, Projects, and Materials was published. 
The Standards are intended to guide evaluations of programs, projects or ■ 
materials in' elementary, secondary, higher, or adult education. The intended 
audience includes persons who commission, conduct, or use evaluations, 
especially teachers, administrators, evaluators, curriculum specialists, 
school, board members, legislators, counselors, leaders of educational 
associations, and parents. 

During the development of the Standards , the Joint Committee consi^erw! the 
•suggestion that referenced be included. After a good deal of discussion, 
the Joint Conmit tee decided against including references, because it was. 
felt that the standards should stand on their own, that references could 
auicklv become dated, and that references might be inappropriately taken as 
a view specifically endorsed by the Joint Committee. ■ The idea of a separate 
bibliography specifically keyed to th'e Standard? seemed a reasonable com- 
. promise that would help satisfy an expressed need for background reading on. 
the topics covered by the .Standards . This bibliography is intended to help 
users of the Standards to identify, literature that includes in-depth informa- 
tion about the issues covered in each of the standards. It is also the 
first of a number of derivative documents and training aides that are being 
developed to supplement the Standards and enhance their utility. 



^American Association of School Administrators, American Educational 
Research Association, American Federation of Teachers, American P^sonnel 
and Guidance Association, American Psychological Association, Association 
for Supervision and Curriculum Development,- Council tor American Private 
'E'ducaticn. Education.Connission of the States, National Association of 
Elementary Schoql Principals, National Council on Measurement in Education, 
National Education Association, National School Boards Association. The 
joint Conmittee in 1981 reconstituted itself. The current Sponsoring 
Organizations and representatives are listed on the preceding page. 

■« ' 

^Joint Conmittee on Standards for Educational Evaluation, Standards 
for Evaluations of Educational Program s. Projects, and Materials. New York: 
McGraw-Hill Book Company,, 1981. 
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The ERIC Clearinghouse on Tests, Measurement, and Evaluation was pleased 
to develop this bibliography with the cooperation of the Joint Committee on 
Standards for Educational Evaluation, Inc. The effort to produce a bibliog- 
raphy began when Robert Carlson of the University of Vermont developed an 
initial draft for the Joint Committee wh'ile he was a visiting ^choUr at the 
• Evaluation Center at Western Michigan University- Muriel Katzenmeyer, a 
research assistant at the Center, added entries to the Carlson draft, -ERIC 
began its work on the bibliography with" a computerized literature search of 
the ERIC database from 1966 through mid-1981. The references of the documents 
and articles identified by fhis search were checked for additional relevant 
citations. Finally, the entries from the ERIC draft were compared with those * 
from the Carlson-Katzenmeyer drafj:, and non-overlapping citations were added. 
At this point, annotations were prepared for all items in the bibliography* 

This annotated bibliography was submitted to the Joint Cotnnittee for re- 
view.. Additions and deletions suggested by them were coordinated by Larry 
Braskamp and Carol Tittle, members of tlje , Joint Cotnnittee, and Robert Rodosky, 
Staff Director for the Joint Committee. They were assisted by Paul Mayberry, 
a graduate student at the University of Illinois.,, These suggestions were 
, incorporated in the final revision of the bibliography. 

The main body of the bibliography is organized to match the Standards ^ 
chapter by chapter. Each entry* begins with the aescriptor for each standard 
V (e.g,. Evaluation Impact), and its definition. Following the descriptor and 
' definition, the annotated references appear in alphabetical order by author- 
For those references in the Educational Resources Information Center (ERIC) ^ 
system, the ERIC accession number is also listed. The Committee decided to 
include only the moFt current references, except for those they consider 
"classic." 

Each reference is listed only once, and closely re:jated sections are 
cross referenced". Citations referring to more than one, of the standards are 
included in a separate section labeled "General Monographs and Textbooks. 
The final section of the publication is an aut^ior index. 

.We woul^ like to thank all the people associated with the preparation of 
this bibliography: Robert Carlson and Muriel Katzenmeyer developed an early ^ 
draft; Kathryn- Hecht of the University of Alaska and Leonard Cahen of 
Arizona State University shared ejctehsive evaluation course bibliographies; 
and James -Sanders of the Evaluation Network provided additional course bib- 

/ liographies that- were "obtained ai a result of a survey by that organization. 

I Past and present Joint Committee members helped by reviewing drafts and adding 
and deleting entries. Irf -addition, the following members were parJ:icularly 
helpful: Larry Braskamp, Henry Brickell, Don Campbell, Ron Carver, Ester 
Diamond, Egon Cuba, Robert .Linn, George Madaus, Bernard McKenna, Lome Shepard, 
Daniel *Stufflebeam and Carol Kehr Tittle. 

We all hope that this bibliography will assist users of the Standards to 
develop in-depth knowledge of the underlying principles and ways of applying 
them. A ■ " * 
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Associate Director 
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Staff Director 
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' A. UTILITY STANDARDS 

1. Audience Identification . Audiences involved in or affected by the 

evaluation should Ve identified, so that their needs can be addressed. 



Hess, Robert J.; Wright, William J. Evaluation Strategies as a Function 
' of Product Development Stages . St. Ann, Mo.: -Central Midwestern 
Regional Educational Lab., 1972. 30p. ED 064 "364 . 

There are issues in curriculum evaluation and stages of product development 
that demand the use of experimental or quasi-experimental designs. To 
counteract criticism of evaluation efforts , an approach to the examination 
of the multiple issues involved in curriculum product evaluation across the 
usual developmental cycle of educJitional products was developed. Curriculum 
products typically move through a developmental sequence comprised of five 

. stages- initial state, hot house (the initial tryout of a prototype product;, 
pilot test, field-test, and public diffusion. Each stage represents a 
milestone in the life of a product. In the course of evaluation, various 
audiences are acquired: the sponsor, the institution, ^the developer, 
consumer representatives, and advisors. There are five major dimensions of 
a comprehensive evaltiation of curriculum products: desirability/feasibility, 
management/procedural cost, product worth, usability, and generalizability. 
Issues relating to the continuation or termination of a program .concern 
statement and fulfillment of objectives, establishing a rationale for the 

.use of particular measuring instruments, determination of whether or not-: 
different effects result from alternative procedures. When the product 
enters the diffusion stage, formative evaluation .is ended and summative 
evaluation should begin. It is pointed out that true summative evaluation 
is consumer protection and is a three tiered process, wherein: the 
product developer establishes the criteria; (2) sbme agency of the federal 
gov'ernment examines the product; and (3.) local education agencies research 
the products. 
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House, Ernest R. The Logic of Evaluative A rgument. CSE Monograph 

Series in Evaluation, 7 . Los Angeles: Center for the ^udy of 
Evaluation, Univ. ot California, 1977. ED 156 719. 

Evaluation is an act of persuasion directed to a specific audience concerning 
the solution of a problem. The process of evaluation is prescribed by the 
nature of knowledge-which is generally complex, always uncertain 
varying degrees), and not always propositional-and by the nature of logic, 
which is always selective. In the process of persuasion one must """^^ain 
who the audience is and find a basis of agreement on premises, both of tacts 
and values, and on presumptions.. Two criteria for evaluation are^ the most 
efficient way to a given end, or the most effective use of available resources. 
Quantitative evaluation methods involve three stages: (1) substantive , 
definition of the problem and its translation into a formal, mathematical 
model: (2) compilation of information in terms of the formal model and its 
forLl, logical analysis; and (3) translation of the formal conclusions back 
into substantive terms. Both formulation and interpretation reqyire good 
intuitive judgment. The ei;aluator and the audience must employ their 
reasoning in a dialogue, and both must assume responsibility since evaluation 
n ne;er\ompletely convincing nor entirely arbitrary. /The logical arguments 
used in two works are discussed. The works-Gene V. Glass' 'Evaluation 
•Skills," and Scriven's reply—are appended. 

McGranahan. Pamela. Implications o f Client Demands for R & E Activities. 
Unpublished. 13p. ED 167 592. ^ 

Potential clients for centralized school district organizations with 
societies, federal and state governments, boards of education, superintendents, 
other administrators, principals, and teachers. A historical «-ew of the 
evaluation literature supports the proposition that some research and evalua 
tion Unit (R & E) clients are served more dir-ctly than others, that this 
service to particular clients is iJi response to their demands; and that 
service to all x:lients is shaped by the demands of .the most direct clients. 
Historically, superintendents were the most directly served alientsi in 
^responiing ^o their administrative needs, R & E units engaged primarily in 
data collection activities. Despite federal and state evaluation requirements 
mandated by the Elementary and Secondary Education ACT, R & E units may still 
be engaged primarily in d^ta collection activities (such as the reporting of 
norm referenced test scores) rather than in evaluation activities. It is 
likely that instructional 'clients such as teachers and project directors 
will be less directly served than administrative clients. 
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Patton Michael Q. The Personal Factor: Identification and Organization 
of Relevant Decision-Makers and Information-Users, I{;.Pf ^on, 
Michael Q., Utilization-Focused Evaluation . Beverly Hills, Calif.. 
^AGEPub., 1978, Chapter 4. 

The first step in the utilization-focused approach to evaluation, is identifi- 
cation and organization of relevant decision-makers for and xnformation-users 
of the evaluation. In a study of factors affecting evaluation utilizat^ion. 
two factors emerged with consistency: the. political considerations factor 
and the personal factor.' The personal factor refers to the P"=«"«^^f ^" 
identifiable" individual or group of people who personally cared about the 
evaluation and the information it generated. Identification of these 
relevant decision-makers and information-users, and determination of th"^ 
Tnformation needs are critical the utilization of the evaluation. Evaluators 
frequently avoid this identification process by themselves becoming the major 
decision-makers for the evaluation, by using the standard J-'^^ J.^^^^- . 
audience" approach, by focusing on the decisions and information rather Chan 
the decision-mak«; and information-users, by deciding that the funj-s of 
the evaluation and/or program are the relevant informatxon-users or by 
targeting evaluations at organizations rather than at individuals.. 

Straton. Ralph G. Ethical Issues in Evaluating Educational Programs 
Studies in Educational Evaluatxon , v3 nl p57-66, Spring ly//. 
EJ 180 463. 

Fiv« broad echical issues which face evaluators in the conduct of evaluation 
Ttudies are discussed: (1) the identification of the f "/^"^^f 
bv the evaluation study; (2) the choice of variables t6 be examined and the 
sources of information to be used in the study; (3)' the technical adequacy 
and cost-effectiveness bf the instruments and procedures to be employed; 
(4) the rights to privacy and confidentiality of subjects and program 
personnel; and C5)^he relationships between the evaluator and program 
sponsors, participants and audiences. 
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A. UTILITY STANDARDS 



1 

Evaluator Credibij-jty . The persons conducting the evaluation shouL 
be both trustworthy and competent to perform the evaluation', .so tha 
their findings achieve maximum credibility and acceptance. 



Ahn, Unhai"R,; And Others. Spectrum -of Qbjecfivi t y-Credibility in 

' Evaluation . Paper presented at the annual meeting of the American 
Educational Research. Associaticm, Washington, D.C, 1975. 17p. ED 
106 367. ' 

Evaluation .roles used in the Department of Research andi Development in Che 
Cincinnati Wlic Schools are identified and described. These include: 
project evaluator; local-school evaluator, independent-program evaluator, 
ekternal evaluator and external auditors.. The'merits of each evaluation 
role are discussed as to its relationship with credibility, objectivity, 
independence and usefulness. The basis for judging the merits- of each 
"evaluation role with regard to the above four criteria are: (1) types of 
decisions to be made; and (2) safeguards to maximize each of the four 
criteria. ^ 



Braskamp, Larry A,; And Others. The Credibility of a Local Educational 
Program Evaluation Report:- Author Source and Client Audience 
Characteristics. American Educational Research Journal , vl5 n3 
pAA 1-450, .Summer 1978. ' * . 

1 

The judged usefulness and objectivity of a simulated evaluation report and 
client agreement with the report's recommendations were examined as functioi^ 
of the evaluator' s simulated professional background, e.g., researcher, 
"evaluator," or "art educator," and the client's organizational role status 
(teachers or administrators). The results suggest that source and audience 
characteristics influence client ratings of the evaluator but do not effect 
changes in agreement with the evaluator's recommendations. 
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Gurel, Lee. The Human Side of Evaluating Human Services Programs: 

Problems and Proipect*. In'Guttentag, Marcia; Struening, Elmer L. 
(Eds.). Handbook of Svaluation Research. Volume 2 . Beverly Hills: 
SAGE Pub., 1975. Chapter 2. 

The -thesis presented .?re- is that organizationaL^context, structural 
constraint's and requirement*, and interpersonal relationships have 
profound consequences for the succe^ or failure of evaluative activity. 
Four considerations related to the context within which the manager and 
evaluator interact are examined: (l) the conflicting superordinate 
organizational goals to which the manager and evaluator subscribe; U; 
the stereotype of scientific omnipotence; (3) the extension of rigorous 
evaluation to areas of public service only recently considered exempt 
from external scrutiny; .and (4) the recourse to evaly<Stion as a pancreas 
for programs in failing health. Within this context, four areas of 
manager/evaluatOL interaction are potential sources of friction: U; 
identifying program objectives, rationale, and procedures; (2) differing 
motivations for evaluation; (3) demands on the operating staff; and (i*) 
the use of rigorous evaluation designs. 

« 

Millman, jason. Selecting Educational Researchers and Evaluators. 
TM Report 48 . Princeton, NJ.: ERIC Clearinghouse on Tests, 
Measurement, and Evaluation, December 1975. 15p. ED, 117 191. 

■ / . 

Aimed at those individuals who are in a position to hire or promote edu- 
cational researchers or evaluators, this paper provides some practical 
suggestioES for assessing thtese personnel. Selection of a research or 
evaLatioS (R & E) fitm is not treated separately from the task of hiring 
an individual; the qulality of work done by a firm depends largely-^n the 
TeoSl who do the job. Much consideration should be giVen to specifying ^ 
?oS description, aid requirements. The value of R & E competencies depends 
upon the specific tasks expected to be performed. A - synthesis of the 
- e??orts of a task force of the American Educational Research Association to 
identify.educational R & E competencies groups under 25 general tasks is 
included in the document. In an .effort to identify a universe of evaluation 
JompeJenc es, Stufflebeam and Bunda produced approximately 250 ^te- 8"??° 
under eight 4ajor categories. The categories and examples of corresponding 
self-assessment items are also included. Several strategies for assessing 
whether an individual possesses the competencies needed for a specific job 
are considered. These include discussions of certification, formal training, 
fcpstinz R & E output, bibliographic and academic characteristics, and 
membership in special professional associations and directories having more 
stringent entry requirements than presently exist. 
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Newman, Dianna L. ; And Others. Communication Theory and the Utilization of 
Efaluation. New Directions for Program Evaluation , n5 p29-35, IS/SO. 
EJ 229 191. • 



The process of reporting evaluation results is described in trrms of communi- 
cations theory. Results of several simulated studies of factors which 
influence the Credibility of eValuators g.id reports are summarized. Jhe 
implications of the results cited are that: (1) the evaluator s credibility 
can be affected by the evaluator 's title, sex, or source of information; 
credibility and perception of the evaluation are affected by use of J"8?" 
and data, and type of evaluative information presented; and (3) the credibility 
of both "-valuator and report can be affected by the receiver's organizational 
position, professional level and field, and perceived need for evaluation. 

Newman, Warren B. Desirable Qualifications for Personnel Conducting 

Educationa l Program Evaluations and Au dits. Paper^sented at the 
annual meeting of the American Educational Research Association, 
1976. 13p. ED 128 389. ^ 



A study was made of professional qualifications for personnel employed as 
program evaluators and auditors. These qualifications, according to 
operational or theoretical models, are necessary to assure local school 
districts of obtaining the services of competent and etlj^ical personnel. 
Findings of:" (1) a review of the literature; (2) a national survey of 
directors and staffs of rt'search'' and program evaluation departments of 
public schools; (3) a review of representative contractual relations and 
iob qualifications in use; (A) a survey of ten university training 
programs; and (5) a survey of legislators to determine the attributes of 
an evaluation report which make it acceptable as a basis for decision 
making, are reported. Criteria for employment of program evaluators and _ 
auditors are recommended, and the political implications of an accrediting 
process are discussed. 



Sanders, James R. School Professionals and the Evaluation Function. 
Journal of School Psychology , vl6 n4 p301-311. Winter 1978. 

Evaluation is assumed to be an integral part of the professional delivery 
of school services. As such, professionals employed in school systems 
are called upon to define alternative roles they might play m evaluation, 
to consider alternative ways to organize for evaluation, and to focus on 
various objects of evaluation. Listed alternatives were drawn from e";«^8^"8 
literature in school evaluation. Standards suggested for judging school 
evaluation included those addressing accuracy, utility, propriety, and 
feasibility of the evaluation. 



A, UTILITY STANDARDS. 



3. Information Scope and Selection , Information collected should be of 

such scope and selected in such ways as to address pertinent questions 
about the object of the evaluation and be responsive to the needs and 
interests of Specified audiences. 



Craig, Marilyn Martin. Assessing the Effectiveness of a Framework 

for the Identification of Information Needs in Program Evaluation . 
Paper presented at the annual meeting of the American Educational 
Research Association, 1979/ 2Sp. ED 174 688. . 



The development and testing oAthe effectiveness of a model for the 
identification of information ^eeds in prograa evaluation are discussed. 
More than 200 subcategories of information needs wore divided into three 
major categories: history, conception, planning aqd development of the 
program; operation — the ongoing processes of the program; and impact 
program results. Five major variables were investigated: (1) number of 
information needs identified; (2) type of information needs identified; 
(3) adaptability to varied evaluation situations; (4) subject ratings 
on importance and priority; and (5) comprehensiveness of the framework. 
Results verified the framework's potential as an effective tool to aid 
evaluators in exploring evaluation situations, in broadening the scope of 
evaluation studies, and in increasing the impact of those studies. 
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Hayman, John; And Others.. On Aggregation. Gener alization, and Utility 
Educational Evaluation .' Unpublished. 1979. 25p. ED 174 667. 

The cross-levels hypothesis is prea«nfced as an, explanation for program evalu- 
ation failures. It states that 6he usefulness of evaluation data as feedback 
for decision making varies inversely with the number of organizational levels 
between the action the data described, and tfie decisions they are intended to 
influence. To be useful for decision making, evaluation data must meet three 
hierarchical infonnatioi^needs : syntactic, semantic, and behavioral. Syn- . 
tactic errors, evaluators should specify their level of reference—individuals, 
classes districts, states, nations—and realize that aggregating- data 
across levels may confuse relationships among variables. Evaluation da^a 
must be on the same level as decision-maker concerns, to satisfy semantic 
needs. Formal evaluation reports, for example, are not relevant to teachers. 
The behavioral need explains why decision makers are less motivated by eyalu^- 
ation data removed' from their level— politically speaking," this information 
is not perceived as important to their own concerns. The cross-levels 
hypothesis ia strongly supported by these information needs, and. offers an 
alternative to research design or statistical procedures as an explanation 
for program. 



Metfessel, Newton S.; Michael, William B. A Paradigm Involving Multiple 
Criterion Measures for the Evaluation of the Ef f ectrveness of School 
Programs. Educational and ^Psychological Measurement , v27 r.4 pt2 
p931-943. Winter 1967. 

The twofold purpose of this paper is (1) to present an eight-step procedural 
outline of the evaluation process and (2) to furnish a detailed listing of 
multiple criterion measures that may be used in the evaluation of specific 
behavioral objectives. The eight major steps in the evaluation process ar* 
direct and indirect involvement of the total school community as facilitators 
of program evaluations; formation of a cohesive model of broad goals, and 
specific objectives; translation of specific objectives into a communicable 
form applicable to facilitating learning in the school environment; instrumen- 
tation necessary for furnishing measures allowing inferences about program 
effectiveness; periodic observations of behaviors; analysis of data given by 
status and change measures; interpretation of data relative to specific 
objectives and goals; and recommendations culminating i-n furtl}er implemen- 
tation, modification.^, and revisions of broad goals and specif ic 'objectives. 
The measures which can be used to collect data include measurefs of student, 
teacher and community behaviors, collected through standardi^sed tests, 
informal instruments, and other means. > 
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Ott, JackM. ; And Others. -Taxonomy of Administrative Information 

Needs: An Aid to Educational Planning and Evaluation. Educational 
Technology . vl3 n5 p29-3!i, May 1973. EJ 079 052. 

Since the quality ^administrative decisions depends in pa-rt on the informa- 
tion the administrator has, incomplete or wrong information will be reflected 
in the -decisions made.' Thus, information needs must be anticipated in order 
that the gathering of that information may be planned. A nine-stage decision 
process is the basis for this taxonomy of information needs. At each stage 
of the decision process, the necessary information is specified. According, 
to the taxonomy, the evaluation team is responsible primarily for locating 
present or potential inconsistencies and' presenting them along with their 
1>robable causes and effects to administrators. It is assumed that the admin- 
istrator will make the subdecisions that are involved in the decision Process, 
such as establishing criteria for judging alternatives or designing potential 
innovations. (The taxonomy is also available as ED ^44 423). 

■ ■ r 
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A. UTILITY STANDARDS 



4. Valuational Interpretation . The perspectives, procedures, anti 

rafonale used to interpret the findings should be carefully described, 
so that the bases for value judgments are clear. 



AndWson,'scarvia B.; Ball, Samuel. The Profession and Prac tice of Program 
Evaluation . San Francisco: Jossey-Bass, 1978. Pages 110-164. 

This section on ethics and values in evaluation includes three chapters. 
The first argues that it is worthwhile for the evaluator to make explicit, 
in as honest and open a way' as possible, his or her values. The second 
notes that Che political-economic context of an evaluation also introduces 
bias, and 'this context must be taken into consideration as the evaluatQr 
forms relationships within and outside of the" progVam being evaluated. ^ 
The final chapteY explores" the ethnical responsibi iVtiflS of the various 
players in an evaluation setting. ^ 



Apple. Michael W. The Process and Ideology of Valuing in Educational 
Settings. In Apple, Michael W.; And Others (Eds.), Educational 
Evaluation: Analysis and Responsibility . Berkeley, Calif.: McCu-tchan, 
1974. Chapter 1 . 

Evaluation is a process of social valuing: it involves one or more groups 
of people assigning values to activities, goals, and procedures done by 
others, such as students. All too often, an evaluation is used to legiti- 
mate an educator's own common \«nse activity rather than to challenge it. 
Evaluation expertise often setv^s as an administrative procedure that is 
relatively ineffective in bringing about; significant changes m educational 



relatively 

processes. *»- — — - — -- , . , mc^ 

decision, institutional evaluation (assessment of the quality ot lite 

students experience in schools) is often ignored 



processes. Because the choice of what one is toassess is itself a^valuative 
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Berk. Ronald A.; Rosii. Peter H. Doi^g Good or Worse: Evaluation _ 
■ Research Politically Re-Exam£ned. , Social Problems. v23 n3 p337 349, 
February 1976. EJ 142 091. 

This paper argues that all evaluation research must necessarily rest on signif- 
icant mSr^l and political .value judgmen^. These and other methodological 
fLJors in turn affect social problem definitional P'^°"-",^V^jn;;;«3,, 
ongoing social programs. Moreover-, evaluation research '•"•P^^^'^^y .^"J""" 
particular ideological perspectives and therefore has broader ^-P^J" 
for social change. Despite these serious weaknesses, however, evaluation 
research may play a progressive role if one s prepared 
designs that capitalize on inevitable value judgments, rather than ignore 



them* 



Berlak. Haroia. Values. Goals. Public Policy fnd Educational Evaluation. 
Review of Educational Research . v40 n2 .p261-278. 1970. 

An Evaluation may focus on programatic or on public P°li^y 'l""'^""'' 
criteria may be used to identify public policy issues: (1) Does the P"8"™ 
alter the -pLer relationship between the citizen and the ^^^^ 
« nerson's status or power within the social systen? (3) Does it increase or 
decrease soc a ?or political tensions? or (4) Does it effect a -^ange in the 
Sf-concept of the individual? Public policy and programatic outcomes may 
?e intend^?, unintended and anticipated, or uninten ed -dunant expat 
This diversity of outcomes" raises the boundary problem, i .e, . the evaiuator 
lust 'derer^i^'e wh\ich outcomes to study.. The expert must set ^-"darie for 
a given evaluatioS task, and the determination of whether he or she wiii 
describe! recommel^'judgment criteria; or. render a judgment depends upon 
Ser the issue is%rimarily a public policy issue or a programatic 



issue* 




-13- 



Eichelberger, R. Tony. Multiple Stakeholders a nd Evaluation. Paper 

presented at the annual meeting ^of the American Educational Research 
Association, 1978. 20p. ED 164 565. 

Evaluations occur within a political decision-making milieu, where multiple - 
stakeholders are contending for limited funds. Given the subjective basis 
of empirical information, different conclusions or recommendations about a 
program may result from different- Ideological , theoretical, and disciplinary 
perspectives. The logic behind, the interpretation of results, and the 
assumptions necessary for such interpretations, must be specified and 
explained to facilitate the most appropriate use of an evaljiation. Because 
of the complexity of many statistical techniques presently used, much work 
is rte'eded to identify 'what assumptions must be met for meaningful and useful 
interpretations oi results in « specific decision-making situation. The 
rationales for both the inclusion "^nd the' exclusion of the variables to be 
considered in an evaluation should be made explicit. ^The problem of obtaining 
a matched cSjtro3. gt^up is oftetx nearly impossible., The relationship between 
.?hrstaSseiXl^a^sis and the evaluation question is often based on enuous 
assumptions. The evaluation of Project Follow Through is used to exemplify 
these problems. 



Gorry, G. Anthony; Goodrich, Thelma Jean. On the Role of Values in 
Program Evaluation. Evaluation Quarterly , v2 n4 p5bl-:)/2, 
Novemljer 1978. EJ 193 492. 

When participants with varied })ackground and interests join 
activity, their different viewpoints may make the eyaluation of the activity 
more difficult. The emphasis placed on different kinds of °""«°%f^y„^^"", 
Rreatly among project participants, causing them to disagree over the worth of 
Se componenfs'of'the program, irrespective of the technical merit^ o he 
evaluation of these components. An experience evaluating a multidi^ciplinary 
biomedical rese'arch center illustrates the influence of values on program 
evaluation. 



Gross, Alan L. Funding Education Projects: Applying Decision Theory 
to the Problem. In Abramson, T.A.; And Others. (Eds .) , Handbook of 
Vocational Education Evaluation . Beverly Hills, Calif.: SAGE 
Pub., 1979. 

Two decision theories (Multiattribute Utility Theor.y and Bayesian Decision , 
Theory) that have been employed by educational evaluators to d^""°" 
makers in taaklng funding decisions are' descriDeo, Both approaches are lUus 
trated in «;eros of hypothetical-, although realistic, examples. 
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Guttentag, Marcia. Subjectivity and Its Use in Eyaluation Research. 
, Evaluation; A Forum for Human Servic e- Decision-Makers, vl n2 
p60-65, 1973. ' 

Edwards'. multi-attribute utilities model for evaluation is described, and 
its application in an Office of Child Develbpment evaluation is d^^ussed. 
^he moLl. which quantises decision-makers' values, is based on decision 
theory, a^d on the belief that the inherent Subjectivity of decision-making 
makes classical experimental designs not very useful. 



House. Ernest. Context ^T^ustif ication. In J"? 

Eds.). Bevond the Game: A Reade r m Educati onal Evaluatio n. 

London: Macmilian Eduction, 1977. Section 4, unapcer 9. ^ 



A distinction is made between the.-"context of valuation" and the "context 
tf u ificaJion." The context of valuation involves the -^-jj-^ 
derived from the genesis of the evaluation, and includes all those motiva 
Sfb ases values, attitudes and pressures from which the evaluation 
aiose The context of justification involves the attempt to justify the 
JindLs Utilizing scientific methodology in the context of justification 
fiaS er;ne o mtSiL bias, but not eliminate it. Since all biases cannot 
be eliminated, it is essential that the scientist reveal the values on which 
his or her research is based. 



House Ernest R. The Conscience of Educational Evaluation. Teachers 

Collie Record. v73 n3 p405-414. February 1972. (Also reprinted m 
• SousrEruest R . (Kd.). School Evaluatipn: The P olitics and Process, 
Berkeley: McCuCchan. 1973. Chapter^^. ) 

A variety of evaluation problems are discussed. Fitst. there is no real 
delani a^ong teachers anS administrators for evaluating fheir own programs, 
unless the evaluation has some direct value for them. In this context, 
eJatuat^s can be used for defense of a program or for °^ 
program. It is useful to distinguish between the context of valuation (the 
TaJue s ant derived from the genesis of the evaluation)^ and the con ext of 
ju ?!fication (an attempt to justify the find ngs Using s"en ific 
Methodology and making valuations explicit will allow evaluations to be as 
unbiased Is possible. A final check on the evaluator's valuations and - 
biases is the interests of other people. ^ 
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House, Ernest R. Justice in^valuation. In Glass, Gene V. (Ed.), 

Evaluation Studies Reylev Annual, Volume 1 , Beverly Hills, Cfelif . : 
SAGE Pub., 1976. Chapter 3. 

The prevalent conception of justice in evaluation is based on utilitarian 
ethics, i.e". the best endeavor is thdt which produces the greatest good 
fot the greatest number. Rawls' conception of " justice-as-fairneJrs' is 
proposed as an alternativ,e. In this paradigm, each person is presumed to 
have nonnegotiable rights which cannot be bargained away no matter how it 
affects the gt^od. The justice of several specific evaluation schema is 
reviewe^. 



Johnson, Mauritz. The Locus of Value Judgments in Educational Program 
Evaluation. Studies in Educational Evaluation , v5 n2, pl09-lZ/, 
1979. EJ 211 901. , . 

The purpose of this article is to examine the various decisions associated 
with evaluation, provide some sort of structure and terminology for^ tnem, 
and then determine where (within this structure) the crucial value judgments 
lie Four typ'es of decisions 'can be distinguished, based, on whether they are 
internal or external, and instrumental or consumraatory: authorizing decisions, 
consequential decisions, procedural decisions, and evaluation decisions. 
Amom; the various procedural decisions that must be made in designing and 
executing an evaluation are four that require value judg-aents: determination 
, of criteria, criterial weights, standards, and rules for applying standards. 
A comprehensive definition of evaluation explicitly incorporates fact, value, 
and purpose: evaluation is (1) a judgment of the inherent or instrumental 
worth, (2) of some educational'entity or. process (evaluand), for the 

purpose of enlightening an anticipated decision-making process, (4) arrived 
at by establishing explicit absolute or relative standards, (5) pertaining 
to relevant Criteria or attributes of th6 evaluand, (6) that have been 
weighted in accordance with their perceived corttribution to the evaluand s 
overall worth, (7) and applying the standards, according to appropriate 
rules, (8) to a full"and accu'rage description of the evaluand, and (9) based 
on reliable observation pertinent to the criteria. 



t 
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Krathwohl.- David R. The Evaluator aS NegotUtions Facilitator-Fact 

Finder. Educational Rv«l..«t:ion and Po licy Analysis , v2 n2 p25 34, 
■ March-April 1980. EJ 229 182. 

Values are involved in every evaluation; the problem is to JJl^^ 
are beneficial values,- beneficially applied, and so perceived by the sponsor 
and relevant audiences. The fact that what is viewed 

oerson or group may npt be so viewed by another, makes clear the difficulty 
of trying fo get evaluations accepted and used when we concentrae solely on 
technical aspects. When audiences disagree on what is benefxcial there 
must be negotiation to reach an agreement on how the evaluation can be made 
Zst mutually beneficial. It is anly as this aspect of evaluation is under 
:?:od and -resolved by the parties interested n it that l^^^^i.^. 
will be perceived as acceptable and extensively used by them in the decision 
making process. 

'\ 

Krathwohl, David R. The Myth of Value-Free Evaluation. Educational 

Evaluation and Policy Analysis , v2 nl p37,-46, January-February i980. 

Values are and must be involved in evaluation. The choice of evaluation 
as a useful process, the definition of its role, what is studied, how 
resources are allocated, all involve valpe judgments. The problem is one 
of determining what is "beneficial prejudice" m any given instance. 
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Kunkel. Richard C. ; Tucker,' Susan A. A Perception-Based Model 0^1^ f 

p^.^.. .^^nni A Values Oriented Theory . Paper presented at the annual 

"meeting, of the American Educational Research Association, ly//. 
23p. ED 152 809. 



Personnel at Saint Louis University's Department of Education have been 
theorizing, researching, and enlarging an approach to P'^^S'^- ^^^^J^ ^^^^^ 
that focuses heavily on the placQ of values in making judgments. This work 
oHginallyst^mmed from general curriculum evaluative theory developed by 
jmTs t McElhi^ney and Richard C. Kunkel. The content of the the°ry ^ 
presented here contains: arguments for a theory of evaluation with explicit 
5uali?y criteria; some quality criteria currently part of ^he perception- 
based model; a statement of the theory and operational paradigm; some 
nolemica developing as the theory and. operational paradigm; some polemics 
SeieToping as the fheory is being applied; and a brief section summarizing 
the Sv'a applications. In the perception-based model of eva uation 
proposed.' certain value criteria are not open to negotiation xn the sense 
Zl aloig with accepting the evaluator personally. J^^.P"^S„^"J^rpSi„,33 
must accept five quality criteria inherent in the model: holism, helpfulness 
Toward program improvement, evaluator vulnerability, acceptance of both 
.•harr(oCf^vable) and "soft" (subjective) data sources, and °^ 
planning a program's future. Additional quality criteria are negotiated 
with individual audiences. 



t 
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Lincoln, Yvonna S.; Cuba, Egon G.» The Distincti on Between Merit and 

■W orth in Evaluation . Paper presented at the annual meeting of the 
Evaluation Netwprk, 1979. 21p. ED 183 574. 

Valuing in valuation ancompasse? two distinct senses of the word, denoted 
hy the terms nSerit ,and worth. Merit may be defined as an entity s inherent, 
intrinsic, context-free value, while an entity's worth is defined as its ■ 
contextually determined, place-bound value. Determining an entity s tterit 
may take place whenever a number of experts are assembled. Worth can only 
be determined by .viewing the entity in operation or on site. Thus, yni^e 
jnerit may be determined in ahy number of ways, worth can be determined only 
by intensive field studies on site. And field studies often call for 
naturalistic, not scientific, approaches. Although it would seem Chat merit 
and worth are identical to formative and summative dimensions, "they are 
orthogonal. It is therefore possible to create a 2 X 2 table and generate 
four distinct types of evaluation: formative merit evaluation, formative 
worth evaluation, summative merit evaluation, and summative worth evaluation. 
Each of the four types of evaluation serves distinctly different purposes 
and is addressed to different audiences apd stakeholders. 

Messick, Samuel. The Standard Problem:" Meaning and Values in Measurement 
,and Evaluation. Airerican Psychologist . v30 nlO p955-966, October 1975. 
"•'EJ 125 292. 

• 

The term "standard" in the title of this article is intended not only in . 
its^ coLon dictionary meaning of "something established for use as a rule 
ir basS of comparison in measuring or judging capacity, qf "tity. content, 
extent, value, quality, etc.," but also in its more general dictionary 
meaning of "something used by general agreement to determine ^^ether or ndt 
a thing is as it should be." Accordingly, this article deals not only with 
questions of meaning but also with questions of values m both measurement 



and evaluation. 



ERIC 



-18- 



Myrdal, Gunnar. Obiectivity in Social Resea rch; The 1967 Wiatner Lecture. 
St. VincenfCbiiege. Latrobe. Pennsylvania . New xork: fancneon 
Books, 1969. 

The most fundamental methodological problems facing, the social scientist are: 
What is objectivity. ,and how can the fesearcher attain objectivity in trying 
^ find out the fac^^and the causal relationships between facts? The logical 
means available for protecting ourselves from biases f",^"^^^^/^^"^-^ '° 
raise the valuations actually determining our theoretical as well as our 
practical research to full awareness; to scrutinize them from the PO^^it of 
view of relevance, significance, and' feasibility in the society under study 
^nransfom them into specific value premises for research, and to determine 
approach and define concepts in terms of a set of value premises which have 
beeh explicitly stated. 



Scriven Michael. The Concept pf Evaluation. In Apple, Michael W. J 

Zd Others (Eds.). P....>^^nn.l Evaluation; Analysis and Responsibility. 
Berkeley, Calif.: McCutchan, 1974. Chapter 3. : ^ 

Evaluation involves making value judgments. The ultimate problem «^out evalu- 
ation is where thd values come from. Value is a complicated theoretical turn 
^h«^ imDlics and. follows from various combinations of desires, needs, and per 
Lmance LaJukion should be thought of as a process of "mpres sing complex 
da™ in he vise^ of the,e contextual constraints so as to squeeze out the water 
«nS leave behind the meaty residue of directed information ttiat is a value 
^TaIIIT tTse vatue judgments are nearly always implicitly comparative if 
Joc'SpUciSy coIjlratiJefnnd a clear recognition of this leads to important 
practical inprovements in the utility of value judgments. 

Sjoberg^ Gideon. Politics, Ethics and Evaluation Research. In Guttentag, 
Marcia; Struening, Elmer L. (Eds.), Handbook of Evaluation Research. 
Volume 2 . Beverly Hills: SAGE Pub. , 1975. Chapter 3. 

Research design should be reconceptualized to tafe account of social factors 
that structure' the research from its inception on through the analysis^of the 
findings. The relationship between the researcher" as a variable in the 
res8«ch design and theory building is most clearly seen when he impact of 
thl researcher's dssaoptions about human nature and* social reality upon the 
Research process is recognized". Although content, they also have a responsi- 
bility to science and to the principle of human dignity to recognize the 
broader political and ethical implications of their efforts. 
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Smith, Nick L. Sources of Values Influencing Educ ational Evaluation. 
Research, E Valuation. Development Paper Seri es No. /. Portland, 
Oreg • — Office of Research and Evaluation Services, Northwest Regional 
Educational Lab., May 1977. 39p. ED 161 8,89. 



/ With the theory that social and personal values influence the conduct of 
evaluation studies in education, the author discusses the impact of two 
majo- sources of such values: contextual factors, including political, 
socipl and organizational influences; and the terminology, models, and 
persbnal values of evaluators. Alternative purposes for an evaluation . 
study are discussed and illustrated. In addition, values hidden in termi- 
nology, value-laden evalua^on models, an evaluators' personal values and an 
illustration of evaluator j^les are treated. The benefits and problems of 
values are discussed and-^bur means of clarifying values m evaluation >^ork 
are outlined. The first approach suggests that all relevant value positions 
need to be identified and stated publicly. The second -approach emphasized 
the need to clarify the evaluatOr's role in the assessment process. Is 
he/she describing the program, recommending evaluation ctiteria, or 
an actual judgment of worth? Tnrough identifying his/^er role, the evaluator 
can choose to de-emphasize his or her personal values. The third JPP^^^h 
BuLests explicitly incorporatitlg opposed values into evaluation studies by 
conLting comparative analysis. The fourth approach reflect., attempts to 
search out conflicting value positions to insure an appreciation of the full 
range. of potentially influential values. 



-20- 



A, UTILITY STANDARDS 



s Rpoort Clarity The evaluation report should describe the object being 
Sated and^l ts context, and the purposes, procedures, -"d findings of 
the Watuation, so that the audiences will readi y understand what was 
done! why it Jone. ^.hat information was obtained, what conclusions 
were'drawn, and what recommendations were made. 

Brarer Gary L,; Mazza. Paul .\e Level of Analysis and the ^-evel of 
' %;esentati;n Are iot the Wme. Fdnc.Honal Eva uatxon and PoUcZ 
Analysis , vl n3 pl05-10< May-June 1979, EJ 211 827. 

Suggestions are made on effective presentations by °^ J^ve"" ' 

studies to audiences v! o are not statisticians. Examples of effective . 
studies ^uai analogies in presenting statistics; graphs 

Tr^oi^or " prSeJLJibnf s:;maries \o highlight findings; cone se reports 
SLeJ on a lefe^i^on newscast style; and judicial use of statistics. 
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■Brown, Robert D.; And Others. Evaluator Credibility as a Function of Report 
Style: Do Jargon «nd Data Make a Difference? Evaluation Quarterly v2 
n2 p331-341, May 1978. (A revised version is available as ED 1J7 J80.; 

1 

The impact of professional jargon and data-based statements in evaluation 
reports on audience responsec to an evaluation report and an eyaluator s 
reconnendations were examined. .Subjects read one of four evaluation reports 
about testing ai^d grading procedures in a <*chool program The reports 
varied in thl amount of jargon and data used to justify the recommendations. 
Ninety-five liigh school teachers and administrators read one of four short 
reporL. each containing one of the following types of statements: ( ) J^"" 
loaded/objective; (2) jargon-free, objective; (3) jargon-loaded, sub.iective 
and (4) jargon-free, subjective. . The jargon-lwaded reports were rated as more 
technical than the argoJ-free reports. The least difficult format was the 
4rgo"free subjectiva report and the most difficult was the jargon-loaded 
^biective report. TW subjective reports were rated as more P"ctical and 
thriargon-loadea subjective reports were rated as less believeable than 
iareJn-free objective repor^'^. There were no differences in reactions to 
'the recommendations of the evaluator. The results suggest that the impact 
of an evaluation report depends upon the style in which it is written. 



House, Ernest R. Coherence and Credibility: The Aesthetics of Evaluation. 
Educational Evaluation and Policy Analysis , vl n5 p5-17, September- 
bctober 1979. EJ 215 210.. 

Evaluation studies .are discussed in terms of aesthetic and literary 
qualities. Concepts such as imagery, coherence, credibility dramatic 
structure, mode of presentation, and story line, are analyzed m relati 
to evaluate documents. 



ion 



Popham, W. James.. Reporting Evaluation Results. In Popham, W. James, 
Educational Evaluation. Englewood Cliffs, N.J. : Prentice-Hall, 



Educ 

1975. Chapter 12 



Consildejrable attention must be given to the procedures employed to "port 
the JesSl-ts oi an educational evaluation. Most often, the evaluator supplies 
evidence to be used by others who make decisions, and so should assume a 
responsive orientation to these decision makers. A variety of techniques 
can'be ised to prepare the final report; for example, preparation of a work 
evaluation report," use of a diversity of reporting mechanisms use ofd^Jfer 
ential depth within a written report, use of adversary reporting techniques 
lT. l\ co^unlcation. specialists, summarization of the results, P^^-J"- 
of /^review copy of the report to those individuals whose program is being 
evaluated* 
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Roberts, Sarah. Communicating Evaluatio n Results. Module 12. Palo 
Alto. Calif.": American Institutes for Research in the Behavioral 
sciences, 1978. 87p. ED 181 345. (Paper copy available only from 
National Consortium Project, American Institutes for Research, P.O. 
Box 1113, Palo" Alto, CA 94302, $3.40.) 

This module is the twelvth in a series on developing p comprehensive career J. 
Guidance program at the high school level, designed to aid guidance personnel 
fe p'sible for developing'student-focused programs.' The f/^ -f,^^^ ^ 

is to help users develop the skills needed no produce an effective evaluation 
report in terms of content, format, level of sophistication, accuracy, and 
organizaiton. The module format consists of an overview, goals, objectives, 
Zlune, time schedule, glossary, readings, skill f-^^P-f^-J-^ ^f^' 
bibliography. A coordinator's guide is also included with detailed in 
sSuiJionS for presenting the module in a workshop setting, as well as the 
facilUator's roU. and functions, and the criteria used m assessing the 
participants' achievement of module objectives. 

Wolf Richard M. Data Analysis and Reporting Consideratiojis in Evaluation, 
'in Popham, W. James (Ed.), Evaluation in Education; Current 
Applications . Berkeley. Calif.: HcCutchan. 1974. Chapter 4. 



The -oath- from a collection of observations and measurements to a setfof 
^^rranted conclusions is fraught with hazards. This chapter describ^ the path 
and offers some guidance on how to negotiate it. It also discusses presenting 
results in a way that can be understood by nontechnically trained persons. It 
Ihou eiable Se reader to better identify and classify each variable in a 
study in terms of its status and scale of °" i.^t 

about the data to be analyzed; identify the stages of ^rea ment of ^ata elect 
an appropriate statistical procedure; and present the re.sults of a statistical 
analysis in a way that can be understood by teachers, administrators, school 
board members, and parents. 
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A. UTILITY STANDARDS 

Report Dissemination . Evaluation findings should be disseminated to 
clients and other right-to-know audiences, so that they can assess 
and use the findings. 



Ball, Samuel; Anderson, Scarvia B. Dissemination, Communication, and 

Utilization. Education and Urban Society , v9 n4 pA51-470, August 1977. 

Dissemination involves more than just telling the world (ot some subsection 
of it) what an evaluation has concluded, ir should involve informing other|^ 
about the evaluation plans, procedures, and latfer its findings. A number c^f_^ 
different audiences should be included in the evaluation plan. The communi- 
cations network for dissemination should include the evaluator, evaluation 
staff, program staff, .and program participants. These groups should be m 
close communication throughout the evaluation. Finally, if an evaluation is 
td be utilized, fvaluators must be advocates of their results, active in 
bringing them -to. the attention of others, and willing to identify publicly 
...any- policy and practical implications 

Patton, Michael Quinn. The Meanings of Evaluation Data: Analysis, 

Interpretation, Dissemination, and Utilization. In Patton, Michael 
Quinn, Utilization-Focused. Evaluation . Beverly Hills, Calif.: SAGE 
Pub., 1978. Chapter 11. 

Evaluation- research is ultimately a personal, perceptual, and interpretive 
approach to establishing the effectiveness of, -human service activities. To 
increase its utilization: (I) present the data in such a way that decision 
makers can decipher and interpret findings for themselves; (2) discuss and 
negotiate the format, style, and organization of final reports with those 
who will be the primary. users of each report; (3) make dissemination efforts 
a matter for negotiation and cooperation betyeen decision makers and evaluators 
• as they work together "to make study findings relevant and meaningful to 
various larger audiences; and (4) .personalize evaluation reports identifying 
both the evaluators who wrote the reports and the decision makers for whom they 
were written. ' , 
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atevens, W. F. ; TornaCzky, L. G.' The Dissemination of Evaluation:' An 
Experiment. Evaluation Review , v4 n3 p339-54, June 1980. 

^ The utilization of program evaluation methodology in human service agencies 

. was reviewed from the perspective of organizational contingency theory. 

Adoption of program evaluation was seen as an innovation which would arouse 
' uncertainty in an organization. A 2x2 factorial experiment, with a sample 
of 37 drug abuse programs, was conducted to\est two hypotheses: (1) group 
■ consultations with sLff would produce moie. innovation adoption than private 
consultations with a program director; and (2) on-site consultations with . 
face-to-face interactions would produce more innovation, adoption than 
telephone consultations. Results indicated strong supiJbrt for the first 
hypothesis, and more ambiguous support for the second. 
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UTILITY STANDARDS 



Report Timeliness . Release of reports should be timely, so that 
audiences can- best use the reported information. 



Anderson, Scarvia B. Dissemination of Evaluation Results. In Anderson, 
Scarvia B.; And 'Others, Encyclopedia of Educa tional Evaluation; 
Concepts and Techniques for EvaluatinR Education and Training 
Prog rams , ^an Francisco: Jossey-Bass, 1975. Pages 130-132. 

Dissemination includes the issues of who should get the results, what, 
kinds of results should be reported for what purposes, and when and in 
what form results should be reported. , • . 



1 
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A. UTILITY STANDARDS 



/ 



8 Eval uation Impact . Evaluations should be planned and conducted in ways 
that encourage follow-through by members of the audiences. 



Agarwala-Rogers, Rehka. Why Is Evaluation Research Not f ^^^f '^'^ J" , 
Guttentag, Marcia (Ed.). Evaluation Studies Review Annual. Volume^- 
Beverly Hills, Calif.: SAGE Pub. , 1977. Chapter 16. _ - 

b^tweln program sfaff and evaluators. lack of emphasis on providing solu- 
tionHo problems, overemphasis on reporting negative findings, and problems 
of feeSSac^and t meliness of results. Suggestions for ;;"=V««=^"f " 
?Ion o-f evaluation results include use of an evaluator who is an insider to 
the organization, involvement of ptdgrflm staff m the evaluation, and 
TrovxTon of liaison individual, or ins^^ti tut ions ^o translate needs into 
evaluation research and evaluation research into practice. 



T„„. AnM nthPrs Increasine the llsp of Evatuation In formation: 
i- in "nt^TiH ^ San Antonio.> Tex.: Education 

S^rvlcl grntef Regipn 20..February l.hO . Paper presented at he annual 
.eting of the Southwest Educational Research Association, 1980. l*p. 



mee 

ED 185 OAO. 



:'i!";l«:„rlo °U =valuatio/dat. to change th.lr ongoing educational 
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Bro-wn, Robert D.; Braskamp. Larry A. Sunmary: Coninon Themes and a , 
Checklist. New Directions for Program Evaluation , n5 p91-97, lyeu. 
/ EJ 229 197. ' . 

■ ^ 

Six co;mon themes related to evaluation utilization as represented 
papers in this issue of tlew Directions for Program Evaluation are 
the definition of utilization, the itmnediate concern of . evaluators, the active 
rSte o ^valuators in enhancing utilization, the "levance of valuation in- 
formation, relationship between evaluator and intended audiences, and .the 
im^Jtanc; of the communication process. A 50-item Ut lization Enhancement 
Checklist is presented, and covers five areas: determing evaluator role, 
unSersJanding organizational context; planning; conducting the evaluation; 
and conmunic at ing' evaluative information. 



Ciarlo. James A. Utilizing Evaluatiftn; Co ncepts and Measurement 

Techniaues. S AGE Research Progress Series in Eva luation. Volume^. 
" Beverly HiUs, Calif.; S AGE^Pub., in cooperation with the Eval^^tion 
Research Society, 1981. ". '~' 

This Collection focuses on utilization of the kind of information known aa 
evaluation: whether program people absorb such information, And how. when, 
Zdllll, if anything, they then do with respect to the programs they operate. 
Srol wliss offers a taxonomy of methods to study different aspects of utili- 
S"on air lists their strengths and weaknesses'. John .Stevenson describes 
an approach to assessing evaluation utilization in human-eryice agencies. 
Ross Conner raises the issue of what organizational levels should be included 
^"g "^^ondents being asked about utilization.- Jud th I^-"- and^^^^ 
Werner discuss the utilization of consultants' suggestions ; for P"g"^ 
-improvement. Cathy Anderson.- James Ciarlo; 'f^^lin. 
■ addition of effective utilization, or a change m emotional state or filing 
about programs, to other types already identified by investigators. Finally, 
Donald Pelz and Jo Anne Horsley discuss the utilization of program-relevant 
' research* . . 

\ 

Cox; G. Managerial Style: 'implications for Jhe Utilization of Evaluation 
Information. ' Evaluation Quarterly , vl n3 p499-508, 1977. 

One of the central problems' with program evaluation is th^ f'^^aiiSroroclsses 
that- results *re not utilized as fully as tossible in decision-making processes. 
. flcrihat a -imilar problem exists, in a wide range of ^-^'^^^^^"XZ, 
. ' . situations suggests that the source of the problem is not^primarily methodolog 
"ar^e «ffcle "draws on Mintzberg^s model of manager behaviors, and then 
ir^; some .inferences ^s to how utilization would proceed and how it might be 
increased. 
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Davis, floward R. ; Salasin, Susan E. The Utilization of Evaluation, 
in Struening, Elmer L, ; Guttentag, Marcia Eds.) Handbook of. 
Evaluation Research. Volume 1 . Beverly ^HiUs, Calxf.: SAGE Pub., 
1975. Chapter 20. 

•< 

After reviewing a variety of SfcT^R^^^^ 
models, the authors present the human action model, or A ^^^^ORY Technique 
This paradigm is influenced by three concepts: the values of the individual, 
iJganizatior or society; the capacity or ability to perfprm according to a 
selected idea: and prevailing circumstances and timing. The four steps m 
Jht of A VKTOR/ technique (assessment, goal definition, action, and 
follow-through) are described and discussed. 

Granville Arthur C. ; An'd Others. The Impact of ' Evaluation; Lessons 
■ Dra^ f rom the Evaluation of FT^^rEifTT Childhood Education Programs. 

?aTr presented at the annual m eeting of the American Educational 
Research Association. 1978. A5p. ED 166 212. _ ^ 

Five different program evaluations are described to indicate those qualities 
!ilch make kn evaluation effective or not effective. Evaluation e£ ectiveness 
was defiLd as impact on decision making or long-term policy formation, and 
irfluen« upon a Variety , of . audiences. Robert D. M^z described the First 
JSfice P'ojec?. and concluded that the evaluation methodology used to inform 
Dolic^akeJs should be distinct ^om the approach used to improve teaching. 
JohnT L-sVe wSo was associated with the national Home Start Demonstration 
^rogrL. let; that seyeral factors contributed to the value and °^ 
. tibn information: evaluations planned with the program; timely reports, 
rigorou SSper^-ntal design; respect between agency and -a uators; demon- 
strated relationships between process and outcomes; and non-controversial 
"ap^fat o^ the progrL. Prbject de^telopmental V^Lliderthe ' ' 

Arthur C. Granville. Factors affecting evaluation impact included tne 
Tetl Jc. of quantitative- data; sociopolitical -"P^^^^^^^^^^^.^J^^irs d 
tions; and pertinence to evaluation criteria. Allen G. Smith, who discussea . 

Project Follow Through, supported clo,e "lf«^i°-^^P%S: ;::?Unti (M chigan) 
-^^K ^r>A rHrriculum Lawrence J. Schwemhart of the Ypsxl^ntx ^.Micnigany 
K y S^cSoS pJo e« reX.ended intensive on-site .tudies. deten..nat^o„ 
of the feasibility of longitudinal aeodies, and adequate funding. 
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Grobe, Robert P. Evaluation— What's It All About? NASSP Bulletin , 
v62 n422 pl-14, December 1978.. EJ 192 36A'. 

' ■ f * • 

An increased emphasis has been placed;,on planned evaluation du6 to increased 
accountability needs, the large numbef of federally-funded projects, and the 
professional needs of educational administrators for better decision-ma-king 
information. Five problems opposing the effective use of evaluation informa 
tion inqlude: (1) ambiguity of outcomes, (2) deciJi^on-makers 
unfamiliar with data, (3) emotional involvement with projects. (4) trivial 
evaluation requirements of federal government, and (5) expecting precise 
akswers. The basic purpose of evaluations is to provide the administratqr . 
with an information base for decision making., including -well-defined priori- 
ties, budgeting based on need, better planning, more efficient operations, 
more effective selection of special projects, and more state and federal 
funds! Stuff lebeam- 8 Cont.ext-Input-Process-Product (CIPP) model is- used to 
demonstrate techniques to improve evaluation utilization. 
* 

'cuba. Egon G. Problems in Utilizing the .^l^fST^ 
of Research and Development in Education . v3 n3 p42-54. Spring 197:>. 

EJ 118 ;)45. '' " 

This paper delineates some of the more frequently enconntered utilization 
problems. First, conflict may arise between the overt foci of the evalu- 
ation and the covert foci, such as compliance or ratifying a decision 
already made." Second, the evaluation may not meet ^he crit.eria.of a good 
evaluation: internal- validity, external validity, reliability, objectivity, 
relevance, importance, scope, credibility, timeliness, pervasiveness, and 
efficiencj. A third source of difficulty stems from discrepancies between 
Iprogram, plans and actual operations. Innate differences among the many 
audiences entitled to receive the evaluation information may hinder 
utilization. A fifth source of difficulty stems from the rapid pace of 
change, both societal change and change in program mission. A sixth source 
of difficulty may result when the evaluator does not maintain a position of 
integrity wiih regard to the program. Finally, the sociopolitical conText 
of the evaluation may hinder its use. \ 
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Haenn. Joseph F. Reasons Why Evaluations and Tes ting Don t Inform. 

Durham, N.C.: NTS Research Corp., April 1980. Paper presented at 
the annual meeting of the Ametfican Educational Research Association, . 
Boston, April 1980. 26p. ED 187 733. 

A number of organizational, personal and methodological characteristics 
have been identified through the literature which inhibit or can be used to 
facilitate the use of evaluation and testing information. Inhibitors o£ 
information usage include (/rganizational characteristics such as loosely 
coupled and decentralised systems, personal characteristics such as lack ot 
awareness of "interest in the needs and values of evaluation, and methodo- 
logical characteristics of the evaluation. Although a few of these 
characteristics are static and not easily changed, most can be modified 
through the strategies of (l) creating a demand for the utilisation of 
evaluation and testing information, (2) facilitating cooperation between 
evaluation personnel and decision-makers, and (3) improving reporting 
practices. A model of loc'al district use of evaluaticm and testing 
information based, on these characteristics and strategies is presented and 
discussed 



rkann, Floyd; Likert, Rensis.' The Need for Research On the Communication 
y .of Research Results. In Caro. Francis G. (Ed.). Readings in 

^ Evaluation Research . Second Edition. New York: Russell Sage 

Foundation. 1977. 

Based on data collected in the Detroit Edison Company in 1948. four factors 
were identified which are important for securina maximum acceptance and 
utilization of survey results: (1) a high degree of participation and 
personal involvement is important; (2) group forces are important in facili- 
tating attitude changes and redefinitions of situations; (3) it is important ^ 
to recognize the hierarchical structure of an organizatiron; it is also 
essential to understand and utilize the power structure as perceived by the 
members of the organization; and (4) participation inla form of self-analysis, 
is more, likely to be followed by changes than if the /analysis is made by an 
outsider. 
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Hovak» Carl D. An Involvement Approach t o the Evaluation of Local 

Di strict Pr-SFrams . Paper presented at tne annual meecmg^oc^the 
;American Educational Research Association, 1977. 46p. ED IDZ 
819. * 

Ways to intrease the use of educational program evaluation findings throug^ 
the meaningful involvement of potential users (teachers and administrators; 
are discusied. Involvement, as defined in this paper, is generally limited. 
" the opp^t^nity for inpuf to the initial evaluation design and the chance 
.^„,•«„'^^^,„ ^.B^n -ncl Imolementat ion plans prior to the evaluation. There 




for the study. For effective, meaningtui invoivemenL. i....: ^^..ww...^ ^---V" 
should be followed: (1) involve only individuals who can contribute something 
or have a stake in the program; (2) screen out biased inputs; (3) use the 
^olicIteS infrrmation,. (4) involve teachers and administrators in the program 
olanninK and implementation, but do not hold them responsible for the evalu 
a ion (5) do not unnecessarily inconvenience, or overburden the participants; 
^6 J^ep he evaluation planning process open; and (7) ^^^^ ■ y."T.,Ain. 
informed of current progress. The evaluation of the ^'^^--GjlKj^^- "^'^^^S" 
disabilities program used in Lincoln. Nebraska is described in detail. 
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Patton, Michael Q. ; And Others. In Search of Impact; An Analysis. 

of - the Utilization of Federal Hea lth Evaluation Re8ear_ch. Minneapolis. 
Center i^or Social Research and Dept. of Sociology, Univ. of Minnesota, 
1975. A6p. ED 135 938. 

Research on the utilization of evaluations was' based on a ^°ll°r"P °^ ^0 
federal health program evaluations to assess the degree to which the evalu 
ation. had been used to to identify tjhe factors that affected v-rying degrees 
of utilization. Interviews were conducted with project officers or people 

they identified as decision-maker, who ^^^^^^ "^/l^^^^^^^^'^f ^""/^.'^it was 
evaluation reports. Two major themes emerged from the study, ^^"t, it was 
found that much of the evaluation literature has ""'^^"^^^^ 
the kind of impact evaluation research is likely to have. Second, the 
im^rtance of fhe personal factor in evaluation research P«"-"i" J^^^J 
utilization process, ha. been considerably underestimated. The two themes 
«e directly linked. The impact of evaluation research is most °ften 
experienced as a reduction in the uncertainty faced by individual decision 
mikers as they attempt to deal with the complexity of programing reality. 
U ^st Se a«imilated and fitted into a contextual whole. Energetic and 
interested people in government can and do use evaluation research, not for 

decisions w^th'in«.ediate, concrete, and visible impacts, but in a more 
T^tll, clarifying, reinforcing, and reorienting way, .^y^lV^^ors, then might 
do wen to spend less time lamenting their lack of visible impact ">«J°;- 
decisions and more of their time providing relevant information to those key 
pfrsons ih^se thoughts and actions, to a substantial extent, etermme the 
Lneral direction in the evolutionary process of program '1«^«1?P"'«"'= * 
in consciously working with such decision-makers to answer their questions^ 
that the utilization of evaluation research can be enhanced. , - 



Tittle. Carol Kehr; And Others. A Procedure to Link Evaluation and 

Finding Decisions. Educational Eval M«t,-on and Policy Analysis. v3 
n3 p43-53. Hay-June 1981. 

Reoorted in thi^ paper is the development and initial feasibility study of a 
!^rof ororedu^es designed to establish a relationship between evaluation 
"id?ngrand fSnSing dicisionnnaking. The decision-making setting was an 
annual grant program for vocational education administered at the^tate 
u"l The procedure to link evaluation and funding decisions required 
determining tL priority and criterion weights for major predictive and 
outc^e impact variables, and providing estimates of the 

which projects might be described for each of th, ^^P^^J^"!"' ^^^"^^^''^ 
was exLiied by surveying local education f^^^.^f/ ^^i""^ 

variable or to give An indication of future availability of data. .This 
ttuSrdemonstrafed that evaluation findings ^nd funding decisions can be 
linked to make better estimates of both predictive and outcome impact of 
projects. 
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Tittle. ^aroi Kehr. Evaluation and Decision Making; Developing a - 
ulthr^ d to Link Program Fundi ng Decisions and Outcome t.yaLuation. 
•paper presented at the -nnual meeting ottneAmericaft Educational 
Research Association. Toronto, March 1978. I4p. ED 155 219. 

There is a continuing need in evaluation research for the establishment ^ 
of a relationship between evaluation findings and ^a^^"?- ^ 

method iB proposL for a particular situation: annual funding decisions 
?or p?oiec?s i^ i Urge grant program in vocational education. Ch^tcome 
and predictive impact variables were ranked by three groups of decision 
makers or. a pilot study. The groups included the Director of the State 
SeparUnt of EducatioS division responsible for funding decisions the 
supervisors who make funding decisions, and the supervisors from related 
' bureaus who review and contribute to the decision-making process. 
Statements concerning the impact of vocational education programs on 
students, employers, and the State Department of .^^ ""J," 

program ^valSation criteria-were sorted into twelve outcome impact and 
ninf predictive impact statements. Each statement was ranked and ra ed 
for importance liy the decision-makers. Results showed high agreement on 

rsnkine and rating of outcome impact statements, and discrepancies on 
rte predictive impaJJ^tatements. A^alidation study has been designed. 
EvalSators can assist decision-makers in identifying important outcomes; 
and in the process, define the decision to be made, the time when it iS 
made and the data required to link evaluation and decision making. 



Weiss Carol H. Utilization of Evaluation Results. In Weiss, Carol H., 
Evaluation Research: Methods for Assessing Program Effectiveness. 
Englewood Clitfs, N.J:: Prentice Hail, 1972. Chapter 6. 

Five constraints which frequently limit the use of evaluation results are 
discussed: (1) the evaluator's perception of her or his ^ole m the 
evaluation process; (2) the organization's resistance to change; 
(3) "adequate disUmination of results; (4) the gap between evaluation 
findings and clear courses for future action;. and (5) the.tendency of 
much evaluation to show Little or no positive effect. In each case, 
approaches fOr improving utilization are discussed. 
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B. FEASIBILITY STANDARDS 

Practical Pro cedures . The evaluation procedures should be practical, 
so that disruption is kept -to a minimum, and that needed information 
can be "obtained. 



Boruch Robert F. On Common Contentions About Randomized Field Experiments. 
In Glass, Gene V. (Ed.), Bvaluation St udies Review Annual. Volume 1. 
Beverly Hills, Calif.: SAGE Pub., 1976. Pages 158-194. 

The resistance to a randomized comparative experimental design to atiswer the 
impact or effectiveness question about a program is deep-set and vigorously 
rationalized. Critics hold that experiments are impoj,=ible to implement m 
the "rell wokd" they are expens^e and slow, they can be replaced by merely 
statistical adjustment of nonexperimental data, they are unethical or that 
thev ignore individual variance and idiosyncracy . In this paper, Boruch has 
maraaUerthe rebuttals to these criticisms and has avoided reconstructing 
the opposing views as ^easily slain straw men. 



Casoer Paul N : Roecks, Alan L. Practical Pr ogram Evaluation. Paper 
^ p^eseSted ^t the annual meeting of the Southwest Educational Research 
Association, 1980. 36p. ED 182 303. 

The practical side of a program evaluation, as performed at a Texas Education 
service Center, is described. The role of the evaluators, as perceived by the 
users of tSe evaluation, and the procedures for evaluating programs several 
levels away from students who are to feel the effects of the evaluation, are 
discussed. 
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Powers. Donald E.; Alderman, Donald L. Practical Techniques for 

iBtjlementing True 'Experimental Designs. Evaluation Quarterly, 
v3'nl p89-96, February 1979. EJ 200 578, 

It is sometimes possible to apply true experimental designs in field settings 
by taking advantage of the constraints under, which programs °5^«^P«"°^""J 
treatmentr> must operate. In a reaearch study requiring classical treatment 
and control groups, practical-methods for implementing true experimental 
SJoceSures in public schools had to be'devised and applied. /i:hcsa solutions 
to a problem often encountered by evaluators are presented here. 



Wick, John W. on Evaluating a Project: Some Practical Suggestions. 
ItCME Measurement in Education . v6 nl pl-8. Winter 1975. (Also 
available as ED 109 10/). 

Prime indicators for realistic short term/long term project goals are 
budgets'and timetables. Concrete, identifiable ""fll^iJ^^^i 
senaratinK eloquent rhetoric from actual promises. Similarly, an external 
"a?u^;; should be able to separate proposals with intentional -""P"-f - 
tion of funding and goals from those which need further o'^S^""? . 
a Project begins, the evaluator should know whether the data being collected 
anSair^zed wili be used for internal public consumption, external public 
reJations! or both. This may depend on whether the evaluators' P^i-ary 
I?legiance is to the funding agency or to the project. In any ^^l"^"""* 
traditional staff roles and lines of authority should be 
better communication facilitated. Technical expertise and the political 
realities of a system should be reconciled. 



B. FEASIBILITY STANDARDS 

Political Viability . The evaluation should be planned and conducted with 
anticipation of the different positions of various interest groiips^^ 
that their cooperation may be obtained, and so that possible attewprTs by 
any of these groups to curtail evaluation operations or to bias or misapply 
the results can be averted or counteracted. ^ 



Banner, David K.; And Others. The Politics of Evaluation Research. 
In Banner, David K.; And Others, The Politic s of Social Program 
Evaluation. Cambridge, Mass. : Ballinger Pub. , 1975 . Chapter 3. 



Evaluation has explicit political overtones. It is designed to yield con- 
elusions about the worth of a given social action program, and m so doing, it 
is intended to affect the allocation of resources. The purpose of this chapter 
is to examine the major literature in the politics of evaluation with an eye 
toward building a model of the process involved. This "model explores the 
dimension of political interaction in the evaluation of social action programs. 



Brickell Henry M. The Influence of External Political Factors on the 

Role and Methodology of ' Evaluation. In Cook, Thomas D.; And Others 
(Eds.), Evaluation Studies Review Annual. Volume 3 . Beverly Hills, . 
Calif.: SAGE Pub., 1978. Chapters. 

Several examples of external political influences on actual evaluations are 
presented. Five guidelines can be used to cope with such influences: lU^try 
to understand how the client thinks; (2) reassure the client that you £an 
interpret the findings so as to give helpful suggestions for program improve- 
ment- (3) find out what the decision-makers will actually use as criteria tor 
judging the success of the project; (4) try to get a supervisory mechanism 
set up for the evaluation contract that contains a cross-section of all the 
powerful decision makers; and (5) write the report carefully, especially when 
describing shottcomings or placing blame. 
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Caro, Francis G. .Issues in the Evaluation of fo^ial Programs Review of 
Kducational Research , v41 n2 p87-114 . April 1971 . EJ 038 6ii,_ 

This' paper reviews the literature on the use of the concepts and "methods of 
behavioral research in evaluating social programs. The first part of the 
paper is" concerned with basic issues which include definitions approaches . 
to evaluation methodology, -roles- of evaluation iir program '^^^^J^P^^"^ 
distinctions among various forms of research. The sfecond section deals with 
orra«i=^aonal niatters such" as the establishment of th.. evaluative research 
tote, administration of evaluative research, utilization of the ^^f^lf 
evaluation, and implications of client activism for evaluation. This Bection 
also includes a discussion of the basic tensions between evaluative researchers 
and administrators: service vs. research, specificity vs. generality me hods, 
status quo vs. change, explanations of failure, and academic vs. practical 
expeJieSce. ^e third section reviews methodological issu.s m measurement 
and design of evaluation studies. 



Enelert Richard M. : And Others. Politics of Program Evaluation in Large 
^ City School Districts. F.d„cat:ion and Urban Society . v9 n4 p429-450, 
August 1977. EJ 166 999. 

This article explores some general notions about politics, f '^i°"'„J"8e 
Tity districts, and their interrelations. Politics, defined in terms of power, 
iiflueice, policy conflict, and similar concepts, permeate every stage of 
progra^ evaluation. Political forces are influential enough to give rise to 
Ehe evaluation effort and to affect its implementation. At the same time 
program evaluation has .an impact on political activities, "P"" ^^^J i^^JT^^ 
makine At times program evaluators themselves engage in political activities. 
?hese political activities are not necessarily unethical or inappropriate, but 
their existence should be recognized. 



House, Ernest R, The Politics of Evaluation in Higher Education. 
Journal of Higher Education . v45 n8 p618-627, November 1974, 
EJ l07 S33~ 

( 

Analyzed are some of the political problems encountered ""J"^^ J^f^^^lj -^3 
ations in higher educatioa. Liberal arts colleges have their own difficulties 
; which evaluation becomes, entangled with suvival of -f^" --3/"^- 
versities have difficulty using evaluation results because of the diffuse 
nature" of their decision-making. Underlying many problems xs the fact that 
projects are used to promote careers, and even moderate public statements 
aJout them can blemish the personal credentials necessary to career advance- 
ment Within these constraints, the evaluator is necessarily in confl ct to 
the degree he or she discovers flaws. Finally, an operational university 
evaluation system which minimizes some of these problems is cited. 
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Levine, Adeline; Levine, Murray. The Social Context of Evaluative 
Research: A Case Study. Evaluation Quarterly , vl nA p515-5A2, 



November 1977, 



Evaluation take's place in a social context that influences research design, 
-selection of variables, the written report, and the timing of its "lease. 
There are also consequences for program implementors, for those subject to 
the program, and for evaluators.. Evaluations and evaluators may become 
involved in political conflict within the subject system and conflict 
external'to it as well. The present study makes use of archival data to 
illustrate the issues in evaluations of the Gary, plan o.f education that took 
^l-ace between 19U and 1918. Suggestions for confronting political and. 
social realities surrounding evaluation emerge fron an application of 
V-etincepts deriving from the sociology of knowledge. 



Mathis, William. Evaluating: T he Policy Implications. Paper presented 

at the annual meeting of the American Educational Research Association, 
1980. lAp. ED 189 123. . " 

Whether initiated by law, regulation, or administrative direction, evaluation - 
has political purposes. Improvement, the classic purpose of evaluation, is 
most faithfully observed when the importance, funding, and constituency of a 
program ak small. -If something is wrong, or if the program's existence xs 
threatened, evaluation can become a weapon in policy disputes, hiding values 
from constituents in a mystique of scientific inquiry. Purposes may also be 
reflected in the biases of those who initiate and conduct evaluations, m 
the selection of a program and^bjectives of evaluation and m the amount 
of evaluation^finding relative-^o -program funding. Similar y, problems 
inherent to evaluation methods are sources of bias. Traditional quantitative 
measures do not lend themselves to broad and sweeping social programs, such 
TsTlingual education. Finally, evaluation results are -of ten used selectively 
to further political ends. In conclusiflpn, evaluation can be viewed as an 
historical enterprise which seeks to recreate Che past with selected emphases 
- or biases. ' 
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Polemeni, Anthony J. The Politics of Evaluation . Paper presented ^ 

at the annual meeting of th e America n Educat ional Research Association, 



1978, lip. ED 163 OAl. 

• 

At one time the major problems faced by an evaluator involved the best way 
to collect, analyze, and report data. Today, an evaluator's major problems 
concern responses to the evaluatibn report by school, superintendents, 
orincipals. teachers, unions, and parents' groups, , An unyiUingness to 
S?i He the evaluation results; the failure.to cons der evaluation results 
ihen making program decisions; the suppression of evaluation data if they 
adversely affect patronage possibilities;' the demand for gross over-simplif i- 
cation in reporting evaluation results; an inability or "^1"""/° '"J^J^ 
new programs, or to alter old ones based upon evaluative data; and the lack 
of communication between evaluators and field personnel are discussed as 
problems resulting from the political forces which influence evaluation. 
The role ofi laymen, Supervisory personnel, project managers, classroom 
teachers, unions, the media, and parents in the politics of evaluation is 
outlined! Politically motivated critics of evaluation ignore the mandated 
necessity of asVessment, and sometimes expect that evaluation agencies will 
be ableUo provide immediate data upon request. The author maintains that . 
the« poYrtical factors result in a lack of funding and facilities necessary 
in the work of a competent evaluator. 



Sroufe. Gerald E. Evaluation and Politics. In Scribner Jay D. 

(Ed ) The Politics of Education; The Sevent y-Sixth Yearbook of the 
NationaTsTcietV for the Study of Education. Part II . Chicago: 
University of Chicago Press.. 1977. 

Politics and evaluation are intimately related. Politics has to do with the 
distribution of stakes within a society or group; evaluation is oriented 
toward improved decision-making, and its goal is a judgement of value, worth 
or merit; .Evaluation is a political resource that can be used to K^^^"" 
the distribution of stakes in education. . Evaluation can be offen"^^^""^^'^ 
taken to alter the existing distribution of stakes) or defensive, (designed 
to thwart an offensive evaluation). Finally, in any study, the politics of 
the individual and the evaluation agency must both be considered. 
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Tuoin. Melvin-M. Politics of Evaluation, In Anderson, Scarvia B.; And 

Others, Encyclopedia of Educational Evaluati on; Concepts and Techniques 
for EvgWing Education and Training ProgramT . S^Francxsco: 
- Jossey-Bass, JLy75. .Pages lUl-iob. 

Political conflicts in program evaluation "can arise over which goals are most 
and which least iApottant, which values are to be preserved and which can be 
sacrificed, what is -the acceptable ratio of cost and effort to g«J" 
achievement, what will determine whether the program has succeeded or failed, 
and who shall make such judgments. The politics or evaiuaLlon r..e.o to a.., 
partisan activities directed at influencing the conduct of evaluation in line 
Kti partisan preferences. So understood, political considerations may and 
almost always do enter into evaluation at every stage. 



Weiss Carol H. Evaluation Research in the Political Context, In 

Struening, Elmer L,; Guttentag. Marcia (Eds.), Handbook of Evaluation 
. Research! Volume 1 . Beverly Hills: SAGE Pub.. 1975. Chapters. 

Evaluation is a rational enterprise that takes place in a political ""text, 
political considerations intrude in three ways: (1) the policies and programs 
with which evaluation deals are the creatures of political decisions, {2) 
because evaluation is undertaken in order to feed into decision-making, its . 
reports enter the political arena; and (3) evaluation, by its very nature, 
makes implicit political statements. 

» 

Wright. William J. Comments on "The Influence of External Polif^al 
' Factors on the Role and Methodology of Evaluation " I" Cook Thomas 
-D • And Others (Eds.), Evaluation' Studies Review Annual. Volume 3. 
Beverly Hills, Calif.: SAGE Pub.^ 1978. Chapter 6. 

» 

The primary issue raised by this paper is the need to examine ways of solving 
the problems resulting from the inevitable intertwinement of politics and 
evalSafion Standards for evaluation should be generated that are consistent 
with the commonly-held values of evaluators. There are two general areas in 
which standards might be generated: contracts (respective responsibilities. 
auSience Restrictions, conflict of interest), and performance (instrumentation 
and sampling, interpretation and reporting, nonperformance}. 
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B. FEASiBiteY Standards 



3. Go-^t Effectiveness . The evaluation should produce information of 
sufficient value to justify the resources expended. 



dost of Edu cational Accountability—A Maryl atid Exploratory Study. 

Denver: Cooperative Accountability Project, Colorado State Dept. of 

Education; Baltimore: Maryland State Dept. of Education, 1974. 62p. 

ED 102 722. ' . ' 

The Maryland State Department of Education participated with the Cooperative 
Accountability Project (CAP) in an exploratory study of the cost-pricmg of 
educational accountability components. The exploratory study was undertaken 
to Setemine the state of the art in cost-pricing of accountability components 
at the state and local educational levels and to enable the organizations .o 
make recommendations about necessary, future research m this field. . Four 
educational accountability components were identified:- goal development and 
implementation, objective development and implementation, status surveying 
of student achievement, and program development. Based on these components, 
a survey instrument was constructed to obtain information from local school 
systems about the costs involved in actually providing information to decision- 
makers. The basic conclusion reached in the survey is that smaller school 
systems will require additional financial aid and technical assistance m 
establishing a comprehensive accountability program. 
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Schriber, Peter E. Coat Benefit Analysis of Comprehens ive AchievemenC 

Monitoring -for-'gI5ii?Sbm~£valuatJ.on . AmherBt::^ School ot baucacion,. 
Univ. of Massachusetts, February 1971. Paper presented at the 
annual- meeting of the National Council on Measurement in Education, 
1971. lip. ED 053 181. 

Comprehensive Achievement Monitoring (CAM) is a systematic procedure of 
constructing and administering longitudinal, criterion-referenced 
CAM has advantages over typical classroom testing of having well-organized 
tests, providing, course evaluation through pretesting, posttesting, and 
retention measurement; producing data for continuous classroom instructional 
management; and being based on a curriculum of behavioral objectives. The 
systematic gathering of comprehensive performance Viata penliits a means of 
establishing a doUar-and-cents cofft analysis for various instructional 
and curricular alternatives. The costs of typical classroom testing and 
CAM are compared. 



ERIC 



9^ d , 



-43r 



C. PROPRIETY STANDARDS 

1. Formal Obligation . Obligations of the formal parties to an eya-luation 
(what is to be done, how, by whom, when) should be agreed to in writing, 
so that these parties are obligated to adhere to all conditions of 
the agreement or formally to renegotiate it. , 



House, Ernest; And Others. An Assessment of the Michigan Accountability 
System . March 1974. 64p. ED 099 21. 

Michigan assumed a leadership role in exploring and applying accountability 
procedures. The purpose of this report is to examine the quality and 
implicatio;s of tSat leadership. Specifically, it assesses tW Michigan 
Accountability System with respect to its educational soyndne.s and u ility 
for Michigan, and with particular emphasis on the assessment component. The 
reporJ prLe^ts both positive and negative findings organized by ^he criteria 
used to assess an accountability program. It also includes a copy of the 
memorandum of agreement between the evaluators and the sponsors ^^^6 Michigan 
Education Association and the National Education Association) As aff 
response to this report is available as ED 111 838; and Stufflebeam s 
response to the staff response is available as ED 163 058. 

A -rnff p~-r---° ^» -y^" ^"V-rt: An Assessment of the Michigan Accountability 
- System^ Lansing; Michigan State.Uept. of Education. May i./a. ^up. 
ED 111 838. 

This response was made to an evaluation of the Michigan Accountability 
system (House, ED 091 82l). Ernest House, Wendel Rivers, and Daniel 
Stufflebeam were contacted by the Michigan Education Association and 
the National Education Association to evaluate the System s ^'^""^lonal 
soundness and utility, with a particular focus on the 
TO some extent, the study produced observations and judgments without 
inaccuracies or emotional exhortations. However, the original report 
contained some inaccuracies, it was not totally unbiased and it^^PP-^^ 
"be based on somewhat unrigorous and hurriedly-gathered information 
Problem areas included observations on goals and objectives, stat« level 
leadership, testing, teacher evaluation, and the ""'^ J °" 

program. Stuff lebeam' s response to the staff response is available as 

ED 163 058. 
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Stufflebeao. Daniel L. A Response to the Michigan Education Department s 
Def ense of their Ac^ntabltlty Syst em. Papgr Tl in Occasional ¥^ 
Series. Kalamazoo: School of Education, Western Michigan Univ., 
■ August 1974. 36p. ED 163 058. (Hard copy available only from the 
Evaluation Center, College of Education, Western Michigan University, 
Kalamazoo, MI 49008.) 

The author responds to reactions by program personnel to an evaluation of 
their program- conducted by him and others. The program was the Michigan 
Accoun?ability System. The program was conducted by the Michigan Depar ment 
of Education and it was evaluated by the author and others who were contracted 
for the work by the Michigan Education Association and the National Education 
Association. - 

The author reviews the history of his agreeing to do the evaluation 
reviews the evaluation findings, presents the program participants reactions 
to the findings, and responds to their reactions. A written set of working 
agreemenifusfd'to govern the study that was agreed to by a 1 parties involved 
with the evaluation (evaluators, sponsor, program personnel) prior to the 
initiation of the work is included. . , 



Weiner, Stephen S.; Rose-Pendleton, M. K. Separate Realities: A Case 
Study of Disagreement in the Design of an Ev aluation. First Task 
FW'Riport . June 15, 1977. 7Ap. ED 152 B14. 

The National Institute of Education (NIE) had commissioned an evaluation 
project of certain postsecondary programs for nontraditional students 
that would involve decision-makers in the core of the design ^^^ivity. 
This report discusses the conflict that emerged between NIE and the Center 
IS^Research and Development in Higher Education (CRDHE) and the f^lure of 
tSeir efforts to resolve the conflict. The terms of the initial agreement 
did not establish a clear priority between the design of an evaluation and 
the necessity of consultation with a decision-maker , nor were the steps 
spelled out in operational terms. Ahe Center's intellectual interests and 
their ties inclined them to questions of direct interest to program managers, 
ZXillelTls primarily Lterested in serving federal fusion-makers 
assessment as to how well specific groups were being served. The existence 
of this fundamental divergence did not become evident until it was too late 
to change the project. Furthermore, there was a clash of styles. CRDHE wa^ 
accuSed to a collegiate styU and relied little on hierarchical Ixnes of 
authority. 
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WriRht. William J.; Worthen, Blaine R. Standards an d Procedures for 
^"^ nl„.1»r..n^ .n, Wem^ntatioa of an;£valuation Oontrapj. ^orlland . 

Oreg.; Northwest Regional Bdu cational Xab. . Oetrober 1V>6. 126p. 

ED 127 341. . ^ 

Intended for individuals and/or Agencies who provide or require evaluation 
lervices this paper attempts to deal with the lack of standards and pro- 
cedures for eJaluSt ion contracts. The first section of the report, Summary 
and^erview of Standards and Procedures fo^ Evaluation Contracting, contains 
alr^f discussion of the use of the proposed standards and procedures, a 
checklist proposed for use in applying the standards procedures, 
instructions 'for use b£ the checklist, and a flowchart which shows the 
in rre "nsSipa and sequence of major events for applying the stan ar s 
and procedures. The second section of the report. Rationale and 
SleJan to the Development of Standards and procedures f^^^^^J^J^^^.^^^ 
Contracting, extends the discussion and rationale referred to in the first 
section and is subdivided into the following subsections;^ ^^^"''p^^iuation 
eSicaJional evaluation; (2)- conceptual issues ^"^^^^^^^J^^^ when ^valua ion 
eaucaLLouai rationale for use oi external evaluation contracts, 1.4; 

i:tLTa fo ;p fy nrcoitr procedures; (5) standards and procedures 

for selecting evaluation contractors; (6) standards and procedures for 
neLtiatioHith an evaluation contractor; (7) standards procedures for 
monitoring an evaluation contract; and (8) applications of, the standards and 
procedures to sample contracts. 
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Wright, William J.; Wof^then, Blaine R. Summary and Overview of Standards 
and P rocedures for Evaluation Contracting , Portland, Oreg : 
Northwest Regional Educational Lab., October" 1975. 25p. ED 127 342. 

The basic thesis of the larger paper from which this condensation is drawn 
is that the use of evaluation contracts is advisable when evaluations are 
CO be conducted by persons external to the institution responsible for the 
program to be evaluated (or. in larger institutions such as large universities, 
by persons external to the unit or department responsible for the program). 
The rationale presented in the later sections has led the authors to propose 
a set of criteria to assist administrator^ and evaluators as they think 
about whether to set up an evalAiati^n- contract and, if so, how to 8° jbout 
it. These criteria are summarized in this paper in the form of a checklist. 
The checklist contains seven subsections which deal respectively with the 
following types of criteria: (1) criteria for determining whether to conduct 
an evaluation; (2) criteria for determining whether to contract with an 
external contractor; (3) criteria to consider when "l^^tijS T'ontract- 

(4) criteria for selecting among procedural options for ^^"8 ^""^"f ' 

(5) criteria to consider when using a request for proposals; (6) "^Jeria 
for use in neogtiating the contract; and (7) criteria for use in monitoring 
Z contJacr^Differfnt sections of the .checklist ViU be useful to different 
individuals and groups for different purposes. A flowchart in which the 
major points of the checklist are translated .into a pictorial sequence of 
events and decisions is also included, 

i 
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C. PROPRIETY STANDARDS 



2. Conflict of Interest . Conflict of interest, frequently unavoidable, 

should be dealt with openly. and honestly, so that it doqs not compromis. 
jche evaluation processes and results. 



Clark WoodrowW., Jr.; Beers, C. David. Ethical Con siderations m the 

Anthropolog ical Evaluation of Educational Programs . Paper presented at 
the annual meeting of the American Educational Research Association, 
1976. 3Ap. ED 129 844. 

In placing the issue of the ethics of using anthropological methods for 
educational evaluation in the context of scientism, anthropology and other 
social sciences are viewed as being in part either qualitative or quanti- 
tative Furthermore, the difference between research and evaluation plades 
the ethnographer in another position in relationship to those studied. Two 
basic categories of ethical considerations are discussed: data gathering, 
including loyalty and employment of the field worker, methodology, and 
confidentiality; and the results of data collection, including the right of 
review, dissemination of findings, and impact of the data. In Section B. 
entitled "The Interaction of Ethics and Method."- some of the ethical issues 
involved in designing a research methodology are dealt with. Comments are 
based on the experience of studying Project Follow Through using a group 
interview technique. 



Molner. Stanley F. Trapped Bedfellows: A Comment on Windle and Neigh er. 
Evaluation and Program Planning , vl n2 pl09-112. 1978. EJ 191 615. 

Molner favorably reviews Windle and Neigher's (1978) paper on ethical 
problems in program evaluation, but suggests that these problems are ^ore 
political than ethical, and that ethical choices cannot be compromised 
whereas political choices can. 
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Scheirer, Mary Ann. Program Participants' Positive Perceptions: 

Psychological Conflict of Interest in Social Program Evaluation. 
Evaluation Quarterly , v2 nl p53-70, February 1978. 

A common dilemma of evaluatixjn researchers, that outcome findings do not 
confirm program administrators' and recipients' perceptions of benefits 
occurring. IB related to a general proposition that participants will have 
positive perceptions of program effects, regardless of behavioral changes 
toward program goals. This phenomenon is shown to occur widely, and to 
be predictable from both behavioral and cognitive social psychological 
theory, but has not been previously recognized explicitly. Implications 
are dr^awn for the policy planning process and for the methodology o£ 
program evaluation. 



Scriven. Michael. Evaluation Bias and Its Control. In Glass, Gene V 

(Ed.), RvflUiation Studies Review Annual. Volume 1. Beverly "^-llf . 
Calif.: SAGE Pub., iy/6. Chapter i>. Ulso available as ED 164 593.; 

The problem of obtaining unbiased information about the merits of a Program 
or product is considered. Some typical cases of bias include divided loyalty 
and the co-option of staff evaluation, and divided loyalty and project moni- 
toring. Two principles are helpful in minimizing bias: (1) no unit should • 
rely entirely on a given subunit for evaluative feedback about that same 
subunit; and (2) since independence is very unstable in an organizational 
structure, provision must be made to insure and continually reinsure the 
independence of the evaluators. Four approaches are capable of upgrading 
JSe Objectivity of evaluation: (1) standardization or routinization of 
qualitative aspects of the procedures; <2) upgrading the training proce- 
dures for evaluators; (3) using the methodology of goal-free evaluation, 

or (4) using an advocate team approach, 

- » 



Sheinfeld, Sherri Nita. The Evaluation Profession in 

Evaluation and Program Planning , vl n2 pll3-115, 1978. EJ 191 616. 

Six values useful for judging the ethical problems in program evaluation 
are: (1) distributive justice; (2) truth seeki^ng; (3) human dignity; • 
(4) sharing; (5) concern for the quality of life; and (6) client loyalty. 
This paper is a comment on Windle and Neigher (1978). 
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Sieber, Joan E.; Sanders, Nancy. Ethical Problems in Program Evaluation: 
Roles, Not Models. Evaluation and Program Planning , vl n2 pll7-lZU, 
1978. EJ 191 617. 

The evaluator must begin with a clear understanding of roles, issues, and 
risks in order to minimize the pressures and occurrence of ethical conflicts 
in program evaluation. A list of such issues is included in this review of 
Windle and Neigher (1978). 



Windle. Charles; Neigher. William. Ethical Problems in Program" Evaluation: 
Advice for Trapped Evaluators. Evaluation and Program Planning, vl 
n2 p97-107, 1978. EJ 191 614. 

Ethical problems in program evaluation are increased "hen conflicting or 
incompatible models are applied concurrently. Three models are ^""^^rj^f ; 
an amelioration model, for a program's own decision-makers; an accoun ability 
model, focusing on public data disclosure; and an advocacy model designed 
to advance the program's interest. Case examples are presented of each, 
Evaluators should consider several general activities to prevent or solve ^ 
ethical problems: clarify roles; build organizational supports; be humble, 
"no fault'! program evaluation; give priority to the amelioration model, 
develop better understanding of ethical aspects of program evaluatipn; and 
design legal supports. . • 
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C PROPRIETY STANDARDS 

Full and Frank Disclosure. Oral and written reports should be open, 
direct, and honest in the ir disclosure of pertinent findings, including 
the limitations of the evaluation. 



Stake Robert E. Evaluating Educational Programmes; The Need and the 
Res ponse . Paris: Organization for Economic Cooperation and 
Development: Centre for Educational Research and Innovation, 1975. 
94p ED 142 565. (Paper copy available only from organization for 
■ Economic Cooperation and Development Publications Center. Suite 

1207, 1750 Pennsylvania Avenue, NW.. Washington. D.C. 20006, §4.50.; 

This survey of recent developments in educational program evaluation is 
intended for persons who commission, implement, direct, or carry out 
evaluation studies. The attitudes of government officials, educators, 
and researchers toward assessment and their own f"j;^°" "^^J^ jy^^e 
discussed, various approaches to evaluation are briefly described the 
author emphasizes informal methods as opposed to standard psychometric 
measures. Instructions for estimating costs of an evaluation project 
are not%rovided, but suggestions for effective use of funds are included. 
Advice is given for evaluators in planning an evaluation study and 
negotiating an agreement with the monitoring officials. "yPO'^'^^^'";, . . 
conversations between an official and a prospective evaluator are included 
as examples that will help the reader start an evaluation properly. 
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C. PROPRIETY STANDARDS 



Public's Right to Know. ' The formal parties to an evaluation should 
respect and assure th 7 public's right to know. ^^^""^^ °^ 

other rented principles and statutes, such as those dealing with 
public safety and the right to privacy. 



1973. Chapter 23. 

\ / 

^ . c ^w. ^hnr.ti>r^ are' (l)'ic is possible and desirable 

institutions; Kl) it is pos^iDi^ nrncpc;sGS- (3) increased involve- 

,„d reasonable access ^° f„?J° "/„7 ^ ;L'e«inrof'ed"c.ao„al endeavors 
ment of people in the f " ,. ? should ensue; (4) some people , 

is in itself desirable, even ;f . i"«"^""'>^ k„e„ ho.;. and'CS) 

„„„ld like to '■«.-"J-f',:''„^^,^^'r„ ^o^ isiln these purpoUs.. it is 
''''"irtharfprur duca ion nfoLtion agency be established to^ 
Tac ult^^tio » rinf™ 

-lc"r~!et:^rb :ra!d the public in under^ 

nirl:at'L''h:irf"° %^r?^"s"pubUr 'rou'prfnrt: a^ th schooU l. 
IJerp^eJiig r:|pfnses fro. the public, as well as helping them assess its 
various priorities. 
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C. PROPRIETY STANDARDS 

5 Rights of Human Subjects . Evaluations" should be designed and conducted, 
* so that the rights and welfare of the human subjects are respected and ^ 
protected. 



Baumrind Diana. Some Thoughts on Ethics of Research: After Reading 
"ilgram's "Behavioral itudy of Obedience." American Psychologist, 
vl9 n6 p421-423, June 1964. 



Certain problems in psychological research require the experimenter to balance 
career and scientific interests . against the interests of prospective subjects. 
SSere experim^^ conditions expose the subject to loss of dignity, or offer 
noting of value to the subject, the experimenter is obliged to consider the 
reason'fe why the subject volunteered and to offer appropriate rewards. The 
experimental objectives 'of the psychologist are seldom incompatible with the 
subject's ongoing state of well-being, provided that the experimenter is 
wUling to Jake the subject's motives and interests into consideration when 
planning the experiment. 



Et hical Standards of Psychologists . Washington, D.C.: American Psychological 
Association, 1953, 1963, lyb^), 1972. 

Specific principles a^ addressed to responsibility, competence, moral and 
feg ttandarSs! misrepresentation, public statements, ^'^-'^^/^^^J' • ^^^^^"'^ 

welfare, client. relationship, impersonal services, announcement of services, 
interprofessional felations, remuneration, test security, test interpretation, 
tSt puWications,%esearch precautions, publication credit, responsibility 
toward organization, and promotional activities. 
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Final Regulations Amending Basic HHS Policy for the Protection of Human 

Research Subjects. Federal Register . v46 nl6 p8366-8392, January 26, 

1981. . ' .. 

• * 

The Department of Health and Human Services amended the policy for the 
protection of human research subjects, substantially reducing the scope of 
Che existing regulatory coverage by exempting broad' categories of research 
which normally present little- or no risk of harm to subjects. 



Kelman, Herbert' C. The Rights of t;he Subject in Social Research: 

An Analysis in Terms of Relative Power and Legitmacy. American 
Psychologist , v27 nil p989-10l6, November 1972. 

The increasing use of social research in American society and its increasing 
relevance to public policy and social decisions have engendered widespread 
concerns about the ethical implications of such research activities. These 
concerns are of two kinds: (1) concerns relating to the P""°^^° °f f 
research, which are exemplified by the issue of invasion of privacy and its 
various ramifications; and (2) concerns relating to the products of. spcial 
research, which focus largely on the fear that social "^^^^^f^ ^^[iJt^ms 
tools for controlling and manipulating human behaviors-. Jhe ethical problems 
surrounding social research can be conceptualized in terms °f P°"^^ 
relationship between the subjects and the scientist or user of the "search. 
These problems should be dealt with by overcoming -or counteracting the 
subject's pover deficifency. 



Shiffer. Lois J. Legal Issues Regarding Sex Bias in the Selection and Use 
of Career Interest Inventories. In Tittle, Carol Kehr; Zytowski, 
D. G. (Eds.), Sex-Fair Interest Measurem ent; Research and. Implications. 
Washington, D.C.: National Institute of EducaCion, ly/B. Pages 135-47. 

This paper sets forth the various sources of law which set requirements on 
guidance test selection and use for school systems and counselors. It focuses 
on laws relevant to test use for careeTr guidance, and indicates steps which 
counselors and teachers can take to select among and use currently available 
tests in a manner which complies with legal requirements. It itidicates what 
interpretative materials are available from test publishers, and what 
materials should be made available directly to students. Finally, jets 
forth suggestions for guarding against bias xn the use of career inventories. 
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Weinberger, JoAnn; Michael, John A. Federal Restrictions cn Educational 

Research: A Status Report on the Privacy Act. Educational Researcher. 
v6 n2 p5-8, February 1977. EJ 156 146. ' 

In a discussion of the Privacy Act of 1974, this article notes that it 
establishes minimal standards for the protection of individual privacy. 
By contrast, educational researchers and the social science community 
generally have a far more restrictive attitude toward the protection of 
individually identifiable data pertaining to research subjects. 



Weinberger, JoAnn; Michael, John. Federal Restrictions on Educational 
Research: A Status Report on the Buckley Amendment and Freedom of 
Information Act. Educational Researcher , v5 nil p3-8, December 
1976. 

This article summarizes the major actions taken by the Federal government 
regarding individual privacy ai/d freedom of information, comments on 
their nature and impact, and highlights current and pending developments. 
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C. PROPRIETY STANDARDS 

Human Interactions . Evalua^ors should respect human dignity and 
worth in their ■ interactions with other persons associated with an 
evaluation. 



Everhart, Robert B. Between Stranger and Friend: Some Consequences 

of "Long Term" Fieldwork in Schools. American Edu cational Research 
Journal . vlA nl pl-15, Winter 1977. EJ 168 848. 

Some major consequences of doing fieldwork in schools over an extended 
period of tiifie are described. Using Powdermaker's distinction of stranger 
and friend," the balance between these two roles and perspectives in terms 
of rone, reciprocity, and receptivity is traced through a description of 
the author's two year study of student life in a junior high school. The 
paper first describes the evolution from stranger to friend by examining 
role relationships between the fieldworker and his or her informants. It 
then focuses upon the problem of the fieldworker having to take on some 
of the characteristics of the groups being studied. Finally the paper 
discusses both the beneficial and counter-productive tendencies of these ^' 

' positions for the fieldworker's receptivity to insights about a- familiar, 
setting. Conclusions center around the dynamic interaction between. 

■ stranger and friend in long-term fieldwork in educational settings. 



Joyce John F. Humanistic Education Through an Analysis of Evaluation 
Practices. Journal of Education , vl57 n3 p39-51, August 1975. 
EJ 125 152. 

UAn analysis of the content, process, and purposes of common evaluation. 

practices has revealed several specific dehumanizing effects on participating 
students and educators. More humanistic, alternative evaluation practices - 
are suggested for each. 
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Rodman. Hyman; Kolodny, Ralph L. Organizational Strains in ^he Re search er- 
Praccitioner Relationship. Iluaan Orgsniz-atxon , v23 n2 pl71-182, 1964. 

(Also in Caro, Francis G. (Ed.). ■R££lill<^^f}"^'^^?; .^^"^f^^^ 
Second Edition. New York: Russell S'age Foundation, is//. Also in 
Gouldner, Alvin; Miller. S. M. (Ed.). Applied Sociology; Opportunities 
and Problems . New York: Free Press. 1965.) 

Potential conflicts between researchers and P^^'^i'^^J"^" ."^ "^^HJ' 
They include the evaluative role of the researcher, the differences 
the way they organize this time, credit and anonymity, patterns of communi 
cation and the relationship between the researcher and the administrator. 
RespoJ^es to resultant strains it^lude denial and displacement, one-way ^ 
humor, and various organizational t^ponses. 

) 
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Ulschak Francis L.; Weiss. Roland G. The Interpersonal Aspectjyof 

itkuaS: A'Transactional Analysis Model for Viewing Evaluator- 
Client Relationships. Educational Technology . vl6 nil plB-O. 
, November 1976. EJ 148 543. 

• 

While it is often recognised that interpersonal problems 
of difficulties for the evaluator, there seems to be a lack of tools 
* avai a^le o aid the evaluator in understanding and dealing with such 
oroblems The purpose of this article is to introduce Transactional 
C?isTs'(Tl? aSd propose it as an explicit and practical model which 
tills this need* 

.* 

Weiss Carol H. The Turbulent Setting of the Action Program. In Weiss, 
r.rn1H. Evaluation Research; Methods for Assessing Program 
Effectiven^ . Englewood Cliffs, N.J.; Prentice-Hall, 1,972. 
Chapter 5. 

Action programs frequently change and evolve during O-he period under 
flJuiy ^houS the evaluator cannot usually control these changes he or 
sie cln document and analyze any significant changes. ^e^^^-^'j^P J^J^ 
program personnel can also cause friction. Possible °f J^^^J?^ 

include personality differences; differences in role; 

definition; conflicting goals, values, if ^^orrctide 
institutional characteristics. Issues that can lead to faction include 
Jata collection; changes in record-keeping procedures; selection of 
progra^ par ici^ants; control groups; feedback of information into he 

rai :g ;i t^^^^^^^^ 

clear role definitions and authority structure. 
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C. PROPRIETY STANDARDS 

7. Balanced Reporting . The evaluation should be complete and fair in 
its presentation o f strengths and weaknesses of the object under 
Investigation, so that strengths can be built upon and problem areas 
addressed. 



p.»p.r^n a Evaluation Reports: A Guide for Authors. Washington, D.C.: 
^Office of Education (DHEW). 1970. 74p. ED U47 002. 

This guide discusses in detail a variety of issues important to the 
oreoafation of a good evaluation report. Main sections are concerned 
SnSTescrib^ng tL context of the program (locale, school system, Btc .) ; 
explaih ng the program (scope, personnel, procedures, etc.); reporting 
tJe evaluation (objectives, sample, measuring and reporting change. . 
Analysis ane presentation of data, etc.); preparing recommendations;- and 
wri ing thrsSmmary, In each section relevant questions referring to 
Zllerl 2U should be considered are.asUed jf.--"^ Idir^^Sd d 
many instances by short exampl^ narratives. Additional aid is provided 
Tn the form of reference lists of standard works, ordered by difficulty 
level on a variety of topics: research methodology, sampling, test 
level, on a variety ui. ^ . onalvsis The Ruide concludes with an 
theory and construction, and data analysis, me k"^" 

example of a complete narrative report. 
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C. PROPRIETY STANDARDS 



Fiscal Responsibility . The evaluator's allocation and expenditure 
"of resources should reflect sound accouT»t ability procedures and 
otherwise bi» prudent and ethically responsible. 



Sladek. Frea E.; Stein. Eugene L. Grants Budgeting and Finance; Getting the 
Most Out of Your Grant Dollar . New York:. Plenum Pub., 1981. 

This text on the management of grVnt money covers the entire grant spending 
process from the decision to apply for a grant to the auditor s f^nal 
approval. It addresses such issues as applying for a grant, negotiating the 
Sest deal, monitoring the fiscal and technical progress of a project, tips 
for cost sharing, funding agency contracts, and cash management. 



D. -ACCURACY STANDARDS 

1 Obi ect Identification . The object of the evaluation (program, 
■project, material) should be sufficiently examined, so that the 
fonn(s) of the object being considered in the evaluation can be 
clearly defined. • 



Fullan, Michael; Pomfret, Alan. Research on Curriculum and I^^C'^^S^ion 
Implementation. Review of Edu cational Research. v47 n2 p335 97. 
Spring 1977. ^EJ 166 914 

Implementation is not simply an extension of.planning and adoption 
ordtesses: it is a phenomenon in its own right. The mam purpose of 
fSis review is to explicate the meaning of implementation and its poten- 
tial determinants by identifying and critically assessing research 
evidence on the process of curriculum and organizational implementation 



Leinhardt. Gaea. Modeling and Measuring Educational Treatment in 
Evaluation. Review of Educational Research . v50 n3 p393-420. 
Fall 1980. EJ 239 573. 



The growth of educational evaluation has brought with it a corresponding 
increase in the desire and need to inclpde information on the nature of 

Se educational treatment that is to be evaluated. However, to date 
there has been no systematic review of how this might be accomplished, 
^his paper explores vays irt which treatment can be ^^-^^^f = ^.^f^^J.^c- 
means of estimating degree of implementation, or by modeling the ins^ruc 
tional domain. The paper als<. reviews approaches to measuring aspects of 
the instructional environment th^t are suggested by the various models 
and methods for combining and analyzing those measures. , 
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Leithwood. Kenneth A,; Montgomery, Deborah J. Evaluating Program 
Implementation. Evaluation Review . v4 n2 pl93-214, April 1980. 



EJ 222 671. 



A methodology for evaluating program implementation .^^^^.'^J'^' 
ments for sSdh a methodology are derived from an analysis of the functions 
to be performed by implementation evaluation, the nature of the program 
beine implemented, ani characteristics of the implementation process, 
Sa features of t-e methodology involve procedures for the deve opment 
of a multidimensional profile of the program as it — 
non- to full implementatibn. The profile then serves as the basis for^ 
instrument development; liata collected through the mstrumen s locate 
program user behavior in relation to the dimensions and levels of 
described by the profiia. Uses of resulting data to serve program manage- 
ment goals are outlined. 



Patton; Michael Quinn. Focusing the Evaluation Question In Patten, 
Michael Quinn, Utilization-F ocused Evaluation. Beverly HiUs, 
Cal i f . : SAGE Pub,, iy/:>. Chapter D. 

' Once relevant decision-makers and information-users ha've ^een identified 
and organized, the second step in utilization-focused «^«l»«t^°" J° „ 
identify and focus the relevant evaluation question, ^rom a utiliza on_ 
of view, the right evaluation question has --"^.^^^^^^^J:^,^^ 
^^ 4* Mssible to bring data to bear on the question, U) tnere is 
^IV.: Tnr os^iile answer to the ^-stion; (3) the ident fied deci^^^ 
makers want information to help answer the '"^o 
need information to help them answer the question; (5) they 5° 
answer the question for themselves, not just for someone else; (6) they 
care about Jhe answer to the question; and (7) they can indicate how they 
vould use the answer to the question. 

« 

Sjogren. BougUs D. Measurement T.chniqjes in Evaluation. BnS±SL 
Educational Research . v40 n2 p301-32O. ly/U. 

The increased comprehensiveness of evaluation efforts and a recognition 
of what is being evaluated has required an expansion of the number and 
type of measurements included in the evaluation. Observation «y«tems 
interaction analysis, matrix sampling, generalizability theory, computer- 
controlled testing. ;nd mastery testing all -have important potential as 
techniques to measure inputs, processes, and outcomes. 
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Steiniaetz, Andres. 



. Program Evaluation vs. Program W^^pvem ent and Some 

Tn.p 1^r«Hons for Training Evalu ators. Paper presented ac "e annual 

meeting of the American Educational Research Association, 1976. iJp. 
ED 128 470. 



To evaluate a specific program means to compare it against a standard 
that specifies what "the program should be like at b specific time. 
Standards may be constructed in three ways: surface 

definition into a model shaped by scientific procedure; deep standards 
include information on the inputs, processes, and outputs for each 
component and subcomponent; and profound standards reach deeper into the 
organization than its task structure and exhaustively covers all dimen 
sions of organizational functioning, and are set by the program staff. 
In this situation then, the role of the evaluator includes expressing and 
explicating the standards set by the program staff, and confronting 
management with the decisions they must make. The evaluator must also 
look at the broader environment of the educational organization, i.e., the 
sociotechnical systems of yhich schools are a part This broader 
perspective implies that the ability to build models the ability to be 
able to gather data relative to a large variety of dif erent Phenomena, 
and to report these data using print, verbal, nonverbal, visual and 
auditory media be included as part of an evaluator s training. 
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accuracy standards 



2. Context Analysis . The context in which the program, project, or 
material exists should be examined in enough detail, so that its 
likely influences on the object can be identified. 



Denny, Terfy. Story Telling and Educational Understandi ng. Paper 

il2 iti Occasional Pap er Series . Kalamazoo: School of Education 
Western Michigan Univ., Novemb er 1978. 29p. ED 170 314. (Paper 
copy available only from the Evaluation Center, Western Michigan 
University, Kalamazoo, MI 49008.) 

Storv telling is defined as a kind of journalistic documentation, based 
on directly observable referents, and used to contribute to an understand- 
ing of educational problems. In the area of educational research, story 
telling is part of the genre which includes case studies, ethnography, and 
ethnology. Storv telling describes an environment in order to communicate a 
general understa"nding of a situation. Fieldwork is the essential ingredient 
in story telling. In educational research, fieldwork consists of familiar- 
izing oneself with local institutions and organizations; talking to local 
officials and citizerts; and particularly, communicating with school personnel 
and working in the school. The keys to successful fieldwork are the ability 
to listen; the mastery of interviewing techniques; acute observation skills, 
and facility at synthesizing information. Story telling, as an ethnographic 
approach, shows. what is happening but does not necessarily reveal causes, 
and may not be the proper evaluation method if the purpose of the study is 
to prescribe change or to determine policy decisions. 
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Scheyer, Patricia T.; Stake, Robert F. A Program's Self-Evaluation 

portfolio. Studies in Educational Evaluation , v2 nl p37-40. Spring 
1976. EJ 168 884. ' 

Though responsive evaluation procedures ;nay appear formidable, this paper 
suggests a way of organizing self-evaluation for projects with only a 
small budget' of money and time. The idea is to establish a file or 
collection of "records or materials which broadly represent the program. 
This portfolio should be a lo6se collection so-that parts of it can be 
differently displayed from time to time. The entries sho^ild reflect the 
program activities, its issues, its valuings, and it compromises. The 
purpose of* the portfolio is to aid and broaden out the ordinary evaluation 
efforts of the program staff. 




D. ACCURACY STANDARDS 



SO 



3 n.e..^K»H P.^r noses and Frocedures . The purposes and procedures of 
the evaluation should be monitore d and described in enough detail, 
that they can be identified and assessed. 



Hager. R. F. Goal Analysis . Belmont, Calif: Fearon Pub. , 1972. 

f 

This book is designed to teach the reader to identify statements that 
describe abstractions and those Chat describe performances; and after 
identifying all important goals, to be able to describe the performances 
that represent achievement of the goals. 

Sand-rs, James R..; Nafziger, Dean H. A_Ba_s^s for Determining the Adec,»acy. 

SrS: „r". ■ ialu on d^agnl? »Sy do need a basis tor Judging Che 
^Luacy- of an evaluation design! Answers to these questions serve to 
:':rZ.. Che. importance o. providing a -»:rc"co rirerl^Ioiff^^lrtant 
r ^dirng ral^^^lon'd::? is'if r"^«e5' Third, a sa.pU design is 

rirra'dX^-oiJ^proiS^ioSii^^^ 

the adequacy of evaluation designs are presented. 
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Stuff lebeam, Daniel t. Meta-Evaluation. Paper #3 in Occasional Paper 

Series . Kalamazoo:' School oi Education, Western Michigan University, 
December 1974. 

Good evaluation reqctires that -evaluation efforts themselves be evaluated. 
Many things can and often do go wrong in evaluation work. Accordingly, it 
is necessary to check evaluations for problems such as bias, technical 
error, administrative difficulties and misuse. Such checks are needed both 
to improve ongoing evaluation activities and to assfess the merits of com- 
pleted evaluation efforts.. This paper presents both a logical structure 
and me-thodological suggestions for evaluating evaluations. Part I analyzes 
background factors and problems associated with meta-evaluation, the need 
for metaevaluation, and summarizes pertinent literature. Suggestions are 
made concerning what criteria should guide the development of meta-eva uation 
methodology. Finally, six classes of problems that jeopardize evaluation 
and need to be addressed meta-evaluation methodology are enumerated. 
Part H is a conceptual response to the first part. If defines and sets 
forth premises for meta-evaluation and presents a logical structure for 
designing meta-evaluation studies. Part III applies the logical structure 
nreaented in the previous section? It contains five meta-evaluation 
designs fiurfor'use in guiding evaluation work, and the fifth for judging 
completed evaluation work. 
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D. ACCURACY STANDARDS 



■ • f. _,.«o The cniirces of information should 

can be assessed. 



Note: 



See also the references for Standards- D5 and D6\ 



C.„pb.H, Donald T. K.apin. Che Da» "/--^^ll^J^I^^I^gST;!' 
Ohio: Grid, 1977. 

The "experimenting society" is proposed as »" "^hrre" 

be scientific. nondx,g«.tic ."""^'f^'t; olv d fofe i ^o^^.y 

.re. however, .any -thodo og.ca^^ Ob e„ o^b.^solve^^,__^^^^^^ 

can be implemented, e.g., cne issues ul l U^A^nntnyfi The ensuing 

surveys, social indicators, and use °J ™ ^„ ? ' islu s statistical 

discussion of related issues groups them as metascientinc 
issues, and political system problems. 



J "rnrriintion" of Data. Eval uation 

Cochran, Bancy. Grandma Moses and l^^' °L 

Quarterly , v2 n3 p363-73, August 1978. EJ 186 186. 

Distortion ot data is caused ^y P-poseful goal-oriented activit^^ 
people who produce data, as well i„ , constructive 

social service delivery systems. /"J""/" „„ positivism it 

motivational ?°»P»-«,;%f ^^f^, 1, c ^Herte "fation'^.ay actually 
i:;:r:rsr^i»o:tt ^ni'de^/e !e the avaiJahUU 

:5ralir;i:L:::rrr:-surgls;r:raU;rn::tfv^r:^'u!i;g program evaluation 
for measuring effects of social intervention. 
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David, Jane L.; Relavin. Sol H. Evaluating Compensatory Education: 

O^er What Period of Time Should Achievement Be Mea|ured? Journal of 
Fducational Measurement . vl5 n2 p91-99, Summer 1978. EJ 1^9 635. 

^ , The goal of compensatory education to increase achievement implies that 
some of this increase stuld be sustained beVond the end of ^e program. 
This paper presents data that allow comparisons between the traditional 
fall-fo-spring evaluation period and a' fall-tb-fall time period. Anal 

sLw'that students in compensatory programs ^^/^^jtcJieJ - 

losses in achievement ov6r the -summer. Therefore, fall to fall achieve 
■ „>ent gains ate smaller .than the traditional fall-to-spring gains. This 
difference in gains can lead to very different conclusions ^^out a 
program's success. If the goal is sustaine^ acnieyement. evaluations 
should be based, at a minimum, on a fall-to-fall time period. 

V ^ \ — 

' .Poynor. Hugh. Selecting Units of Analysis. In ' ^^^^^ °- ["l^ ' 

vL^,..tin ^ Educational Programs and Products Englewood Cliffs, 
N.J.: Educational Technology Pub.. 1974. Chapter 15. 

A orooer unit of analysis is the smallest source of data that is both 
togi°a?Iy aiS s atistital'ly defensible. Both approaches separate pupil 
.ini s from classroom averages, although to different "degrees Simu at.d 
. ,^piricaUdemonsti:ations are used to reveal the importance of choosing 
the proper unit of analysis. • ^ 

i 

S^iwin. Enoch I. Curriculum Evaluation or ^^^P^^^^'j^J^^'^y ; 

in Educatio. ^ Evaluation . v2 nl p41-51. Spring 1976. EJ 168 885. 

Problems associated with current expertise in evaluation are disc';'^"^- 
Since evaluators are not always able to reliably achieve all eve s of 
an evaluation project, these tasks /are categorized into five levels of 
compUxUy T^e author suggests that evaluators should retrench down the 
s'le of c;mplexity of an evaluation until a level is "-^f ^ .f^ 
(1) conclusions arii. reliable across investigators and are scientifically 
defensible. (2) reVults obtained pose minimum threats to personnel, and 
(3) training requirements for evaluators are within reason. A more 
accurate label for such evaluators would be "descriptive inquiry 
specialists." 
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Stake, Robert E. Objectives, Priorities, and Other Judgment Data. 
Review of Educational Research , v40 n2^ pl81-212, April 1970. 

Four kiYtds of da'ta are considered judgment data: personal value-commitments, 
objectives, the priorities given to certain objectives, and standards. In, 
evaluation studies, judgment data should be gathered and analyzed. Surveys, 
scaling,- the Q-technique, the semantic differential, observation, and expert 
review are all methods for gathering judgment data. Though difficult to 
summarize, -judgment data should be reported, possibly m narrative form or 
U8i\ig a profile or matrices.' Finally, the jugrient data should enter into 
decision processes as inputs, not as outputs. 



Tittle, Carol Kehr. Test Bias: Current Methodology and Implications for 
Evaluators. In Abramson, Theodore; And Others (Eds.), Handbook of 
Vocational' Education Evaluation . Bevferly Hills , Calif . : SAGE Pub., 
• 1979. Chapter 20. 



After describing key definitions and requirements in the Uniform Guide- 
lines on Employee. Selection Procedures, this chapter reviews the procedures 
and methods that have been used for examining test and item bias m the 
educational assessment setting, in the absence of an external criterion. 
Finally it presents a series. of recommendations to evaluators, listing 
the data that evaluators should find in test manuals, the data that are 
needed to make the determination that a test is fair for use with particular 
groups, and the procedures evaluators will find useful in minimizing test 
bias in local test development. 



Tittle,, Carol Kehr. Use of Judgmental Methods in Item Bias Studies. 

In Berk, R. A; (Ed.), Handbook of Methods for Detectin g Test Bias. 
Baltimore: Johns Hopkins Univ. Press, m press. 

The renewal of interest in Item bias and the fairness of tests used in 
evaluation has focused attention on the test development process and con- 
struct validity. Judgmental methods used throughout the test development 
process include procedures \o examine stereotyping and fair representation 
of groups. Judgments also provide validity-related evidence: • familiarity 
of Rroups with the nominal content of items and the opportunity to learn 
item content and process (the match or overlap of items with the curriculum 
and the instructional process). Reseaifch and procedure* in these areas are 
described. 
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D. ACCURACY STANDARDS 



5. 



Valid Mea surement . The inforniation gathering instruments and Procedures 
should be chosen o r developed and then implemented m ways that will 
assure that the interpretation arrived at is valid for the given use. 



Note: See also the reference for Standards D4 and D6, 



f 

BerH, Ronald A. '(Ed.) Criterion-Referenced Measur ement; 
Baltimore, Md.: Johns Hopkins Univ. Press, 19B0. 



The State of Art. 



This book is a product of the first annual Johns Hopkins University National 
Siposium on Educational Research, held in Washington D C in Oc^^^^^^^^^ 
1978 It attempts to determine the state of the art of criterion rete.encea. 
i!ILement It includes discussions of content do.a n specification and 
item generation, item and test validity, and reliability. 

Cook. Thomas. "D.; Campbell, Donald T. Quasi;Experimentation: Design and ' 
Analysis Issues for Field Settings . Chicago: Rand-McNally, 1979. 

This book presents some quasi-experimental designs and design features that 
fan be used in many social research settings. Each design is assessed in 
?™ o? four types of validity, with special stress on internal -validit)^ 
TlZull g rer r onclusions sle drawn about the strengths and J-^tations/^ 
of eacfdfsign; emphasis, is also placed on the fact that the relevant , 
threats to valU inference are specific to each research setting. 

« 

Cronbach, Lee J.; And Others. . The Dependability ^^.f ^^^^g^^^^^"'^;:^;^^ ' 
Theory Of Generalizability for Scores a nd Profiles. New York. Johh 

Wiley & Sons, 1972. 

This monograph presents d theory for evaluating the 8!?"^!^ "^'^^^^^ „ 
of test scores and profiles, and scores derived from field observations. 
It contains concrete examples and problems for advanced students m 
measurement theory and research methodology. ^ 
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Cronbach, Lee J. Validity ort Parole: How pan We Go Straight? New 
Dire ctions for Testing and Measurement . n5 p99-108, 1980. 

Presented at the 1979 Educational Testing Service Invitational Conference, 
this article reviews developments in test validation in the past decade. 
As with a scientific theory, . interpretation of a test is going to remain 
open and unsettled, the more so because of the role values play m legal 
and policy actions based on tests. 



Hambleton, Ronald K. ; Eignor, Daniel R. Guidelines for Evaluating 

Criterion-Referenced Tests and Test Manuals. Journal of Educational 
Measu rement , vl5 n4 p321-327, Winter 1978. EJ 198 850 

y, 

A set of guidelin&s for evaluating criterion-referenced tests is presented. 
The guidelines address objectives, test items, administration, test layout, 
reliability, cut-off scores, validity, norms, reporting of test score 
information, and test score interpretations. Additionally, 11 ^^^s 
of e^ant cHterion-referenced tests are evaluated using these guidelines 



Messick, Samuel. Test Validity and the Ethics of Assessment. Ameri^ 
Psychologist , v35 nil pl012-27, November 1980. EJ 235 612. 

Questions of the adequacy of a test as a measure of the characteristic it 

interpreted to assess are answerable on scientific grounds by appraising 
psychometric evidence, especially construct validity. Questions of the 
appropriateness of test use in proposed applications are : 
ethical grounds by appraising potential social consequences of the testing. 
The first set of answers provides an evidential basis for test interpre- 
tation, and the second set provides ,a consequential basis for test use. 
By then considering both the evidential and consequential bases of both 
test interpretation and test use. the roles of evidence -'^^ -^"l^^^"^^^^" 
the overall validation' process «re illuminated, and test validity comes to ■ 
be based on ethical as well as evidential grounds. 
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NunnallY. Jum C; Durham, Robert L. Validity, Reliability and Special 

Problems of Measurement in Evaluation Research. In Struenmg, Elmer 
L ; Guttentag, Marcia (Eds.), Handbook of E valuation Research. 
Volume 1 . Beverly HiUs, Calif.,: SAGE Pub. , 1975. Chapter 10. 

— — ^— — 

\ 

This chapter discusses methods for determining the validity of measures, 
and principles concerning reliability. In one way 

issues discussed in this chapter concern generalizability. Thus, the- 
validity of a predictof test concerns the extent to which one can 
generalize from scores on the test to scores on a criterion variable. 
Reliability concerns the extent .to Which one can/generalize from scores 
on a test to scores on alternative forms of the test. 
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Porter, Andrew C; And Others. Impact on What?: The Importance of the 
P:„^ .n^ nnv.vpri. Research Series No. 2 . East Lansing: Michigan 
State Urtiv., Inst, for Research on Teaching, February 1978. 37p. 
ED 155 215. 

Defining practical significance in program evaluations is a difficult 
measurement problem which can only be solved by an ^^ntimate familiarity 
with the measures on which effects are estimated and their content 
relationship to the program goals. Past attempts to P'^°^;^^ ^^"^"J 
solutions to the size of effect problems have relied on standardized 
indices which can be estimated and reported without any ^^n?"l«Jf °^ 
what was measured. Such efforts are viewed here as steps in the wrong 
direction. Instead, what is called for is a Procedure "hereby the 
content goals of the program, Che content implied by a tfist, and the. 
Interrelationship between the two are made explicit. The Procedure 
should investigate treatment-by-item intWactions and at the same time, 
describe ^he mLsures used so that persons other than the evaluator can 
reach their own decisions about practical significance. Analysis of the 
mathematics sections of four major intermediate leve standardized tests 
with their taxonomies indicated rather substantial differences in content 
tested. It was clear that standardized tests are not well suited to the 
task of estimating item domain by treatment interactions. - 



Shepard, Lorrie. Purposes of Assessment. Studies in Educational 
Evaluation . v5 nl pl3-26. 1979. EJ 210 291, 

Assessment generally refers to large-scale, system-wide measurement 
programs for pupil diagnosis; pupil certification; program eva uation. 
JesLrdh; accountability; resource allocations; or teacher evaluation 
The purpose of assessment should determioie the test content, construction, 
administration; and examinees sampled. Assessment methqds for one purpose 
may be inappropriate for other applications. 
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Standards for Educational and Psychological Tests . Revised Edition. 

Washington, D.C.: American Psychological Association, 1974. Also 

relevant for Standard D6, Reliable Measurement.) 

This document presents standards for test use as well as for test manuals; 
it is intended to guide both test developers and test users. These standards 
apply to any ass^essment procedure, assessment device, or assessment aid. 
They are grouped in three levels: Essential, Very Desirable, and Desirable, 
The standards cover tests, manuals, and reports; reliability and validity; 
and the use of tests. They were prepared by a joint committee of the American 
Psychological Association, the American Educational Research Association, and 
the National Council on Measurement in Education. 



Walker Clinton B. S tandards for Evaluating Criteri on-Referenced Tests. . 

1^8 Angeles: Center for the Study of Evaluatiori, Univ. of Calitornia - 
Los Angeles, January 1978. 33p. ED 179 595. 

/ 

Standards for evaluating criterion-referenced tests ^^"f ^ 

one standards,- grouped in three categories, are discussed. Category One is 
defined as Measurement Properties and is comprised o^ conceptual validity, 
including description of the domain, test item agreement with objectives, 
and item representativeness of the objectives; and field test validity, 
including sensitivity, item uniformity, divergent validity, lack of bias, 
and consistency of scores. Category Two is labelled Ap^roprioteness for 
SLinees! a^is comprised of clarity of instruction; icem review; physical 
format, including layout and legibility; and ease in recordifig answers. 
Category Three is called Practicality, and is composed of adequacy of 
inforaaiion about the test; relevance of items of at ieast two serie. of « 
teaching materials; flexibility, including multilevel testing of objectives, 
alternative test forms; clarity of test admistration directions; scoring, 
recorSkeeping; availability of rules to make instructional decisions based 
on test results; and comparative data on test scores. It is also stated 
that the test buyer must determine the degree of correspondence between the 
objectives of a test package and the objectives of the curriculum to be 
tested. 

Wargo, Michael J.; Green, Donald Ross (Eds.) Achievement Testin;^ of Disadvan- 
taged and Minority Students for Education al Program evaluation. New 
York: ' CTB/McGraw-Hill. 197/. 

This book represents the proceedings of a conference of the same title held 
in Reston. Virgiriig,. in May 1976. The purpose of the conference was to 
iSent fy? define, an'd analyze problems associated with the .use of standardized 
achievement tests on populations of disadvantaged and minority students for 
educational program evaluatipn. 
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b. ACCURACY STANDARDS 



6, Reliable Measurement . The inf onnation-gathering ^^^^ruments and 

procedures should be chosen or developed and then implemented in ways 
that will assure that the information obtained is sufficiently 
reliable for the intended use. 



Note: See also the references for Standards DA and D5 



Cook. Thomas D.; Campbell. Donald T. Quasi-Experimentation: Design and 
Analysis Issues for Field Settings . Chicago: Rand McNally. 1979 . ^ 

This book presents some.quasi-experimental designs .f„^^f,,Jr ^^/^^^/^ 

■ can be used in many social research settings. The designs ^""^ J- J . 
causa! Sypotieses L,out a wide variety of substantive issues both bas.c 
and applied research." Each design is assessed in °^ 

■ vriidi?y--statistical conclusion validity, internal validity, construct 
TAiAlv and external validity-with special stress or. ^^1 . „. 
General conclusions are drawn about strengths and limitations of each design, 
however. e:;ha:L is also placed on the fact that the ° - 
valid inference are specific to each research setting ^^^^ /^^^ ^^^^.^ 
deal with quasi-experimental designs and modes of -nalyzing ^^J^a^^f 

from liem, Anothe? chapter deals with causal inference from designs that 
Uck most of the characteristic features of experimental research. The 

final cS pter states that randomized -P"if?" . "f.^^^^^^Kn '"m lay 
field research, and outlines obstacles to their implementation, some ways 
4f iverco^tng these obstacles, and some ways of recognizing the situations 
when random assignment is most feasible. 
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Cronbach, Lee. J.; Gleser, Goldine C. Interpretation of Reliability and 
Validity Coefficients: Remarks on a Paper by Lord. Journal of 
Educational Psychology , v50 n5 ?230-237, October 1959. 

Most statements describing the usefulness of tests as judged from their 
reliability or validity coefficients assume that a decision is made about 
every person tested, i.e., the persons are divided iVto three classes: 
those whose true scores are greater than a specified criterion score, 
those whose true scores are less than the criterion, and those for "Jom 
neither interpretation may safely be made. This paper differs from Lord s 
in placing emphasis upon the maximum risk of erroneous interpretation 
ratSer than upon the average tisk. The suitability of a test depends not 
only on the reported reliability and validity coefficients, but also on 
the importance of the decisions to be made and on the rules by which the 
scores are to be converted into interpretations. 



Linn, Robert L. ; SUnde, Jeffrey A. The Determination of the Significance 
of Change Between Pre- and Posttesting Periods. Review of Educational 
Research . v47 pl21-l50, Winter 1977. EJ 161 389. 

The major issues that arise in the measurement of change are reviewed 
and, where possible, alternative approaches are discussed. The measurement 
of individual differences is considered fifst. This is followed by a 
discussion of some of the concerns involved in inferring treatment effects 
from group differences. The concluding section discusses accountability 
systems based on student achievement. 



Lord, Frederic M. The Utilization of Unreliable Difference Scores. 
Journal of Educational Psychology . v49 pl50-152, June 1958. 

The purpose of this paper is to call attention t;o a natural way in which 
difference scores haJing relatively low reliability may be (and currently 
aJe) used effectively; and to suggest a method for inferring from the 
reliability coefficient of difference scores their effecti,venfess when 
used as outlined. 



/ 
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Nunnally. Jum C. The Study of Change i" Evaluation Research: Principles 
Concerning Measurement, Experimental Design and Analysis. In 
TrZl7, E1.er L. ; Guttentag Marcia f°:>' 3;-f,;f ° Cha°p e r 6. 

Research. Volume I . Beverly Hills. Calif . : SAGh Pub. , .iy7X unapcer 

This chapter discusses problems which frequently arise in the study of 
Sanee 'reducation research. The issues are discussed m the order that \^ 
thevwoiu occur to the scientist: measurement (constructing measures \ 
expost ^acto'^eactivity of measurement, faking of responses extent of 
:rsurem:nt iroblem. subjective assessments), then "-"^^/^^^^^^^^.^^ 
(experimental and quasi-experimental), and finally statistical analysis. 

Education, "l971. Chapter 13. ^ 
ThP fact that repeated sets of measurements never exactly .duplicate , 

test-score theory. 
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D. ACCURACY STANDARDS 



7 ^ Syste matic Data Control . The data collected, processed, and reported 
in an evaluation should be reviewed and corrected, so that the 
results of the evaluation will not be flawed. 



Ball Samuel. Audit of Evaluation. In Anderson, Scarvia B.; An^ Others, 
'Encyclopedia ^f Pf^..^.'»rinn^T Evalu ation: Concepts and Techniques for 
Evaluating Education an d^r-^ining Programs . San Francisco: Jossey- 
Bass, 1975. Pages 40-42. 

The idea of auditing evaluations came from the educational accountability 
movement that developed in the 19bO's. The actual work of the auditor 
varies somewhat, but can include looking over the evaluation plans, 
monitoring data collection, checking the analyses, reading an early draft 
of the evaluation report, suggesting changes in the report, and approving 
the final version. 



Demaline, Randy E.; Quinn, D. William, H ints for Plannxng ^ and Conducting 
a Survey and a Bibliography of Survey Meth ods^_AidJ _injnstruc^- 
tional Aids Series . Kalama zoo: School of Education, Western ^achi^t.^ 
Univ., April 19797 107o. ED 173 417. 

Methods of planning and administering mail surveys, developing, question- 
naires, and analyzing data are reviewed. Each review 

by an -notaCed list of selected readings. Topics discussed in plapning 
a su..^y include decision-making; survey designs; sampling plans; and 
ethical considerations. Development of instruments and types ^^ti ude 
measures are discussed in the section on survey instruments. The section 
on survey management is concerned with the mechanics of distributing and 
collecting the questionnaires, coding, and checking for e"ors. Data 
analysis focuse. on nonresponse analysis,, computer usage, and the choice 
of statistical methods. A 365-item bibliography and a subject' index 
;eared to the bibliography are appended. 
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Murphy, Richard T. Quality Control. In Anderson, Scarvia B.; And 
Others, Encylopedia of Educational Evaluation: Con cepts and 
Techniqu^for Evaluating Education and Trai ning Programs. San 
Francisco: Jossey-Bass, ly/^). Pages 299-301. 

The methods by which the sample is selected and its quality tested are 
statistical quality-control methods. For quality control m evaluation, 
data collected with tests, interviews, observation techniques, ratings, 
and other methods, laUst be examined to see whether they are J^JgeJ 
sufficiently free of error to be worthy of further analysis. Quality 
control procedures should be used routinely from the very first phase ot 
,dAta collected through to the last stages of analysis. 



Stufflebeam, Daniel L.; And Others. Educational Ev.aluation^nd Decision 
Making . Itasca, 111.: F. E. Peacock, 1971. Pages l/b-197. 

When the delineation of information needs is completed, the evaluator 
must establish a plan to obtain the information. This plan should 
consist of the following areas: colle^ction of data, organization of data, 
and analysis of data. These tasks can be further broken down into work 
units • 



D. ACCURACY STANDARDS 



8 Analysis of Quantitative Information . Quantitative information in 

an evaluation should be appropriately and systematically analyzed to 
ensure supportable interpretations. 



Bentler, Peter M.; Woodward, J. Arthur. Nonexperimental Evaluation 

Research: Contributions of Causal Modeling. In Datta, LoiS-ellen; 
Po^lnff, RnhPrt (Eds.K Improving Evaluations . Beverly Hills: SAGE 
Pub. , 1979. Chapter 6. 

This chapter discusses the .relevance of causal modeling research methodo- 
logies to evaluation research, reviews in a nontechnical manner a series 
of causal modeling techniques for both quantitative and qualitative 
measures, and concludes with an example applying structural equation 
modPls to data from a summer Head Start . program. This nontechnical 
introduction is intended as a first step toward assessing causal modeling 
in evaluation research. 



Bryk. Anthony S.; Weisberg, Hetbeiib I. Use of the Nonequiyalent Control 
Group Design When Subjects Are Growing. Psychological Bulletin, v84 
n5 p950-962, September 1977. 

/ 

In the nonequivalent control group' design, pretest and posttest data on 
both groups are obtained. Statistical methods are used to adjust posttest 
comparisons, based mainly on pretest information. The purpose of this 
article is to consider the adequacy of these methods, from an individual 
erowth perspective. It is concluded that statistical adjustments ajre 
generally inadequate in the face of nonequivalent growth systems across 
treatment groups. 
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Cooley, William W..; And Others. Analyzing Multilevel Data. In Berk, 

Ronald A, (Ed.), Educational Evaluation Metholology : The State of 
the Art . Baltimore: Johns Hopkins Univ. P.rpsB, 1981. Chapter 3. 

* 

The critical i-tesues in the analysis of multilevel data from evaluation 
studies are identified and discussed. Their presentation is restrict^ to 
explanatory observational studies where the statistical methods usually 
involve the analysis of relationships among variables, e.g., multiple 
regression. The review of the issues is organized in four sections: the 
importance of choosing a causal model prior to choosing a method of analysis; 
atfRregation bias; the implications of the variation that might occur in 
within-group coefficients; and general strategies, for analyzing multilevel 
data. 



Cronbach. fee J. Analysis of Covariance in Nonrandom ized Experiments; 

Parameters A ffecting Bias . Stanford . Cbii f . : Ev^iuat-ion Consortium, 
Stanford University, August 1977. ■ 

A model for nonrandom experiments is developed Co evaluate the bias in the- 
adjustments made to compare outcomes in nonequivalent ^'^?"P^ ' ^i^^^^^J^^lT " 
ment made in analysis of covariance depends on the covariate employed.. The 
covariate can be expressed as a weighted combination of an ideal covarxate, 
which determines outcome scores within a treatment group; a discriminant. . 
which determines assignment to treatment group; and irrelevant information. 
The presence of irrelevant information reduces the • „ 

adjustment. When the covariate contains little or no irrelevant information, 

aSus meht may be too large or too small, depending on the correlations 
of the covariate with the discriminant and the ideal covariate. Correction 
procedures now present in the literature cannot be counted on to provide an 
unbiased estimate of the treatment effect.^ * 



Cronbach. Lee J.; Furby, Lita. How Should We Measure "Change - Or Should 
We? p'vcholoeicar Bulletin . v74 nl p68-80, July 1970. Errata. Tsjrcho; 
logical Bulletin , v'lh n3 p^8, September 1970. 

Procedures previously recommended by various authors for the estimation 
of "change" scores, "residual" or "basefree" meaiures of change, and 
other kinds of difference scores are examined. A P'^^"'^"" P^P"^?^^*'^ 
Lord iff extended to obtain more pr-cif- estimates, and an alternative to 
Che Tucker-Damarin-Messick procedure is offered. A consideration of the 
purposes for which change measures have been sought in the past leads to 
a series of recommended procedures which solve research and personnel 
decision problems without estimation of change scores for individuals. 
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Kenny, David A. A Quasi-Experimental Approach to Assessing Treatment 

Effects in the Nonequivalent Control Design. Psychologi cal Bulletin, 
• v82 n3 p345-362, May 1975. 

I 

Four statistical tests of treatment effect are evaluated far the non- 
equivalent control group design: analysis of covariance, analysis of 
cQvariance with reliability correction, raw change score analysis, and 
standardized change score analysis. Given a model of the process of 
selection into treatment groups, the nonequivalent control group design 
can yield interpretable results. 



Marascuilo, Leonard A. Measuring Differendes among Non-Randomized Groups: 
An Epidemiological Model for Identifying Successful School Program. 
J ournal of Experimental Education , vAS nl p50-59, Fall 1979. EJ 220 
353. ' ■ 



It is recommended that the biomedical model of adjusted statistics 
designed to overcome the difficulty investigators face when attempting 
ta randomize subjects be adopted. The adjusted discrepancies between 
croup statistics are considerably smaller than is indicated by inspection 
of raw. or unadjusted, sample valbes. This model provides a way to obtain 
a more accurate estimate of program, success or failure when comparisons 
across classrooms or other units are desirable. 



Porter, Andrew C; Chibucos, Thomas R. Selecting Analysis. Strategies . 
In Borich, "Gary P. (Ed.l, Evaluating Educational Programs and 
Products . Englewood Cliffs, N.J. : Educational Technology Pub. , 
1974.-. Chapter 16. 

* 

Evaluation paradigms are divided into four categories, determined by the 
presence or absence of random assignment, and the use of a pretest or the 
use of some other variable observed antecedent to treatment. Far each 
category of design, the following analysis strategies are considered: 
analysis of covariance using a random covariate, analysis of variance of 
an index of response includihg gain scores as a special case, repeated 
measures analysis of Variance, and analysis of covariance using estimated 
true scores as the cDvari^te* 
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Weisberg. Herbert I. Statistical Adjustments- and Uncontrolled Studies'. 
Psychological Bulletin . v86 n5 pll49-1164, September 1979. 

A variety of problems are related to a lack of experimental control: 
measurement error, unequal growth' rates across groups, and regression 
artifacts, 'in this article it is shown that these problems can all be . 
subsumed under a general conceptual framework, as particular examples of 
model misspecification. The case of linear adjustment (analysis of , 
covariance) is given. special attention. 



Wolf. Richard M. Selecting Appropriate Statistical Methods > I" 

Ronald A. (Ed.), Educational Eva luation Methodoloi^v; The State of 
the Art . Baltimore: 'Johns Hopkins Univ.. fress 1981 . unapter 5. 

Several statistical mcthods\f or analyzing the results of an evaluation study 
are compared. Factors that guide the selection of analysis of variance and 
Zx,Z of covariance are delineated in the fiVst settion. The -J- 
?n s^Lequent sections is the analysis of nonranaomi.ed designs Special 
attention is given to" designs based on comparable. groups and to those based 
on noncomparable groups. , , - ^ 
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ACCURACY .ST/tn)ARDS 



An alysis of Qualitative Information ! Qualitative information in an 
evaluation should be appropriately and systematically analyzed to 
ensure supportable interpretations.. 



Alternative Methodology. In Hamilton, David; And Others (Eds.), Beyond 
the Numbers Game: A Reader in Educational E valuation. London: 
Macmillan Education, 1977. Section 4. 

Alternative evaluation is an eclectic approach, adaptive and responsive 
to the particular learning milieu in which the evaluator is working. 
These are intellectual traditions outside education that illuminative 
evaluators dtaw upon: participant observation in sociology, ethnographic 
field work in social anthropology, literacy criticism, fi Im. documentary , 
historical research, law and clinical psychiatry. A variety of papers 
are presented in this section, covering case studie?, tield work and the 
generation of theory, and the community context of evaluation. 



Becker, Howard S. Problems of Inference and Proof in Participant 
Observation. American Sociological Review , v23 n6 p65Z-b6U, 
December 1958. 

The basifi analytic operations carried on in participant observation are 
descrioed. These stages of analysis are conducted in the field, the 
selection and definition of problems, concepts, and indices; the check 
on the frequency and distribution of phenomena; and the incorporation of 
individual findings into a model of the organization under study. A 
fdurth stage of final analysis involves problems of presentation of 
evidence and proof. 
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Bogdan. Robert; Taylor, Steven J. Introduction to Qualitative Research 
^ ulZ,.: A Pheno;enoloRical Approach to t Ke Social Sciences. New 
York: John Wiley, 1975. ^ 

Over the past decade, there has been growing interest in the sub-jective, 
in meaning, and in common-sense understandings. This book discusses 
qualitative methods. as they relate to the phenomenologica Pe"P"^ive. 
?att on« contains a "how to do it" approach to Participant o^serva on. 
personal documents, open-ended interviews, and examples 
studies. Part two is a discussion of how to present findings, with 
several example reports. • 

Everhart, Robert B. Problems of Doing Fieldwork f g^);"^""'^""' 
Humln Organisation , v34 n2 p205-2I5, Summer 1975. EJ 119 820. 

The use of fieldwork in evaluation presents the fieldworker and Agencies 
connected with educational evaluation with a number _of "^^ij^^ P^^^^^J' 
fouv of which are identified and discussed m this paper. The fitst, the 
identification of the evaluation problem, notes the distinctions between 
the traditional evaluator who defines evaluation problems in an a priori 
manner aid the fieldworker who defines the problem ho Ustically and as a 
resuU o? preliminary fieldwork. The second problem fbcuses upon the 
unclear and changing signals which the fieldworker receives from the 
ageicy sponsor"! the evaluation. A related problem is the -^^^P "^^^ 
of expectations the fieldworker receives from various members in his or her 
role set iSe dissemination of data and consequences of dissemination is 
JSe tSird problem area discussed. Th^ paper concludes with an 
of some compromises necessitated by the fieldworker doing evaluation work. 
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Ferreira, Joseph; Surges, Bill. Collecting Evidence; A Layman s Guide 
to Pa rticipant Observation . Boston: Institute for Responsive 
Education, 1976. 28p. ED 132 715. (Paper copy available only from 
the. Institute for Responsive Education, 704 Commonwealth Avenue, 
Boston, MA 02215.) 

Participant observation is useful as a tool for gathering evidence ^bout . 
processes, circumstances, or other observable conditions. _ A participant/ 
observer is an investigator gathering fvidence. Observations are care- 
fully recorded, prejudgment is scored, fand judgments flow from the 
evidence. In approaching a situation to be investigated, the participant/ 
observer should get a flavor of th6 system of which the situation is a 
part, identify those problems or parts of the system that influence the 
situation under investigation, and -Select the problems or parts that seem 
most important and might 'provide vital evidence." Once the situation is 
chosen, .five types of data are often important: descriptive data about 
setting's, accurate descriptions of actions and behaviors, word-for-word 
statements, traces and wear .spots, and documents. The participant/observer 
is also interested in reliable witnesses and informants. Self-training 
exercises and samjple observations are included. 

I 

Fienberg, Stephen E. The Collection and Analysis of Ethnographic Data in 
Educational Research. Anthropology and Educat ion Quarterly, v8 n2 
p50-57. May 1977. 

The major theme of this paper is that, from a scientific viewpoint, thare 
is no fundamental difference between the two sides of the qualitati.ve/ 
quantitative controversy. The process of statistital inference is 
basically the same for both types of research. Ethnographic researchers 
have pinpointed a major flaw in much educational research: the unit ot 
analysis need not be the same as the apparent unit of sampling. Finally, 
in addition to using multivariate methods to analyze their data, mvesti- 
gators need to begin thinking in terms of large-scale randomized controU-ed 
field trials. 



Filstead. William J. Using Qualitative Methods in Evaluation Research: 
An Illustrative Bibliography. Evaluation Review , v5 n2 p25J-ZbO, 
April 1981. 

This article. briefly describes the ways in which qualitative methods have 
been viewed relative to evaluation research. The topics included in the 
bibliography include: the changing climiite in evaluation research, the 
philosophical and conceptual background behind this approach to research, 
actual evaluation efforts which employed qualitative methods, the use of 
various data gathering techniques, and how one "makes sens?" of these data. 
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Guba Egon G. T^wnr rf a Methodology of Na turalistic Inquiry in Educational 
' Eviluat o n. CSE Monograph Series m Eval uation ,!. Los AHieTFs : 
- Center for the Study Evaluatio n. Univ. ot cITIfornia-Los Angeles. _ 
1978. 97p. ED 164 599. 

i 

Evaluation is viewed as essential t'o decision making and ^ocial policy 
development. Since conventional methods have been disappointing or 

i^I qua " naturalistic inquiry Wl) . ^"^""^^1^1"^"" ' 

in miSimiz ng constraints on antecedent conditions (controls) and on 
output (dependent variables). N/I is phenomenological rather than 
Dofcitivist. It offers alternative strategies for problems when the 
^Jeriment^l approach is implausible. A number of \iew evaluation models 
(sSch as the responsive mo^el. the judicial model, and the connoxaseurship 

O ar ompa?ible with th; approach. Since, there n° ' 
?^ truth N/I must be credible and deal convincingly with standard 
letJodo?;8ical problems such as boundary problems setting the scope of 
inquiry ! focussing problems (establishing and defining categories), and 
3lem of authenticity (reliability, validity, and objectivity). 

iL^rtialitv Is imperilled by conscious or unconscious bias, mcompetjence. 
^nm' or ^rruptibili^y. It is promoted by openness and fairness. 

Krippendorff. Klaus. ^Content Analysis : Xn Introduction to Its Methodology. 
Beverly Hills, Calif.: SAGE Pub., 1980. 

content analysis is an important rese^frch ^-^"i^"^ J" J^! ^^^.^'.^^^l^r" 
The methodology seeks to understand data not as a collection "f Physical 
I^ents but as symbolic phenomena and to approach their ^"^^y"^ 
sively This boJk presents three aspects of content analysis is theory, 
: thid and procedures, and qualitative criteria, ^"/"/""^-f^ ^fj',:, 
brief histo- of content/ analysis is presented as well as a definition that 
d ng cfnte^t a^lysis from Other methods and J -^^.l^^if^^^^ 

■ -ri"\sr?is^^rdesrs^:;r"" rairsrs's^puS! 

To^::; "Jlon^of ir Ungu:;e!: Analytical constructs, computational techni- 
ques and the use of computers. The quality criteria of content analysis 
Te ;e?LbiUty and validity. Suggestions are made as to ^^^v^.^ "^f,^^/^^ 
criteria. Finally, the book concludes with a practical guide for doing 
content analysis. 
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Lofland, John. Analyzing Social Settings ; A Guide to Qualitative 

Observation and Analysis . Belmont, Calif.; Madsworth Pub. i;ompany, 
1971.. 

A positive and detailed set of instructions are presented indicating 
exactly how qualitative observation and analysis are performed. -In the 
first three chapters there is an attempt to spec-ify what qualitative 
analysis is and how it differs from quantitative analysis. There follow 
concrete descriptions of the two basic techniques used by qualitative 
observers in collecting their materials; interviewing and participant- 
observation. Finally, recommendations are made as to how one can store 
and organize materials to facilitate more acute observation, analysis, 
and writing. ^ 



Smith, Louis M. An Evolving Logic of Participant Observation, Educational 
Ethnography., and Other Case Studies. In Schulman, L. C. Ud.;, 
Review of Research in Education , v6, 1978. 

• • 

This chapter -provides a context and logic- for the discussion of educational 
ethnography by making three major points. First, a large body of both 
8ubs?ln?ive snd methodological- literature within this fifeld study traditid^ 
a^eady exists Second, a reflexive overview of ^J-^P^^^or^eiJs 
field work suggests a perspective on methodo^^ogy. Third, the essay presents 
.patterned analysis of this genre of research, considering four major 
^mains: data, descriptive narrative, theoretical, and metatheoretical. 



Smith, Louis. Integr.ating Partic pint Observation into Broader Evaluation 
Strategies. In Hamilton, David; And, Others (Eds.), Beyond the 
Numberf ya;e; "a Reader in Educational Evaluation . London; MacmiUan 
Education, 1977. Section A. "Chapter 6. 

Participant observation can be integrated with other evaluation techniques 
in three different general evaluation strategies; a general structural 
model, a sequential model, and a case-study accumulation model. The 
generk structural model brings together three research strategies, an 
experimental design, a social survey, and participant 
sequential model attempts to cumulate efforts over time "^l^^^^^Jj; 
currently in time. The third strategy is the cummulation of participant 
observer case studies. 
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•'•rend M. G. On the Reconciliation of Qualitative and Quantitative Analyses: 
A Case Study. In Cook, T.; Reichardt, R. (Eds.), Qualitative and 
Quantitative Methods in Evaluation Research . Beverly Hills, Calif . : 
SAGE Pub., 1979. Chapter 4. 



THis paper examines an instance where the analysis of qualitative data fr 
a participant observer produced an explanation that could not be reconcil 
immediately with one based upon quantitative data from the same social 
experiment. The presentation is designed to: (1) give the reader insight 
info the social psychology which operates fn large-scale research efforts; 
(2)'dispel the notion that using multiple methods will lead to s^ 
explanations in an easy, additive fashion; and (3) suggest a way 
in resolving a conflict between two different explanations of th 
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Wolcott, Harry. Criteria for an Ethnographic Approach to Research inSchools. 
Human Organization , v3A n2 pi 11-1 28, Summer 1975.. 

A number of issues 'related to conducting ethnographic research in schools 
are explored under four criterion headings: (1) appropriateness of the 
problem, (2) appropriateness of the ethnographer, (3) appropriateness 
of the research "climate," and (4) appropriateness of e^Tpectations for 
th-e completed study. The purpose of the p-^ger is to air some of the 
.issues that arise between those who do ethnography in educational settings 
and those who commission it, by examining recent experience and customary 
expectations. With the express purpose of fostering further dialogue, 
the author states his own position on many of the issues, but the critical 
function served by the paper is to work toward explicating problems rather 
than necessarily resolving them. 
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ACCURACY STANDAReS- 



10 Just ified Conclusions . The conclusions reached in an evaluation 
should be explicitly justified, so that the audiences can assess 
■thein. 



Note: See also the references for Standards A4 and D8. 



Campbell, Donald T.; Erlebacher, Albert. How Regression Artifacts 

■ in Quasi-Experimental Evaluations can Mistakenly Make Compensatory 
Education Look Harmful. In Struening, Elmer L.; Guttentag Marcia 
(Eds.). Handbook of Evaluation Research. Volume 1 . Beverly Hills: 
SAGE Pub 1975. Chapter 19. (Also in Helmuth, J. (Ed.)', Compensa- 
tory-Education: A National Debate. Volume 3 of The Disadvantaged 
child" New York: Brunner/Mazel, 19/0.) 

Evaluations of compensatory-educational efforts such as Head Start are 
commonly q,uasi-experimental or expost facto. The on^ht 
is made available to the most needy, and the "control" group then sought 
from among the untreated children of the same community Often this 
untreated population is on the average more able than the "experimental 
group. In such a situation the uSual procedures of selection adjustment, 
and analysis produce' systematic biases in the direction of making the 
.- compensatory program, look deleterious. Not only does ""f ^^^"S/"^""^^^^ 
regression artifacts in this directipn, but so does ."'^ly^^^f 
and partial correlation. This essay illustrates with a detail^ example 
why these biases appear. 
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F.»nnes8ey, James. Blending Evidenc e Technique, and Judgment in Educa 

Hnn^l Researcra fprence. Final Report ^^^^^^^ ^ ELM .3 
" Social ReXationg, Johns Hopki ns Univ., November 1976. 298p. ED 1« 

675. 

research can be attributed to the use of an inappropriate -"odel for 
researcn can u ^.^uor oresents and discusses advantages of 

Bayeslan is to fhe sort of data most frequently encountered in 

data analysis. 

« 11 r«n» F Loucks Susan F. A "Developmental Model for Determining 
Research Journal , vl4 n3 p263-276, Sumer 1977. EJ 180 513. 

documentation °f ,"''"5" "^'^^^ i.oUs can be. reliably measured: 
ir„L:Jiirnra:tin.'pf.pa'ra"on -achanical u.... routine, refinement. , 
integration, and renewal. 
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Kennedy, Marcy M. Generalization of Findinlgs from Single Ca se Studies. 
Paper presented at the annual meeting of the American Educational 
Research Association, Toronto, 1978. 26p. ED 155 222. (Available 
in microfiche only.) 

Although single case studies might be useful to evaluators for a variety 
of purposes, there are no generally accepted ways for drawing inferences 
about the generality of findings from a case study. Single case studies 
are defined in this paper as either ^studies of single events, or disag- 
gregated studies of multiple events. The data may be qualitative or 
quantitative, and may be derived from controlled experiments or from 
observation. There are two spans to the bridge of inference. The, 
statistical span connects the experimental sample to a popvilation just 
like that sample. Th'e second span connects the population to a group 
judged to be sufficiently similar. In case law or in clinical practice, 
the judgment of sufficient simlarity--that is, the judgment of the ■ 
appropriateness of the generalization-is made by the user. This appli- 
cation of single case data may also be appropriate -kn educational 
evaluation. 

Lindvall, C. Mauritz. Basic Considerations in A ssessing the Validity of' 
Eval uatiort Designs . Paper presented at the annual meetrmg of the 
"American Educational Research Association, San Francisco, 1979. Z^p. 
ED 170 359. 

Evaluation studies on educational questions attempt to provide answers in 
the form of conclusions or inferences which are derived from the informa- 
tion collected. Valid inferences are a result of careful research design 
for the study, and may be causal, descriptive, value-oriented, or proba-, 
bilistic. Basic steps in designing an evaluation study are suggested. 

(1) developing examples of each type of inference which will result; 

(2) identifying major components of each inference} (3) specifying 

the validity concerns of each component; and (4) planning the study so 
that inferences will be defensible against claims of invalidity. Types 
of validity applicable to evaluation studies include content validity, 
constrict Validity, internal validity, population validity, and ecolcgical 
validity. 
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Porter, Andrew C. ; And Others. Practical Significance in '^"^""'^..gg 
Evaluation. American Educational Rese arch Journal, vl5 n4 p529 5Jy, 
Fall 1978. EJ 200 567. 

Defining practical -significance in program evaluations is a difficult 
Te ^rl^t problem, which can only be solved by an -J-J^^/^i 
with the measures upon which effects, are estimated, f ^^^5"%="^^'3?f,^!?, 
relationship with the goals of the program being evaluated. Past attempts 
to describe the "size of effect" of instriictional programs have char ac- 
't^ri'ical^y relied on" statistical indices that ""f « «f/-^/^Ji^"/i, 
reported without "any knowledge of what was measured.^ Thxs ptactice is 
shown to be misdirected. Instead, what is called for is a Procedure 
whereby the substantive instructional intentions of the program, the 
substantive characteristics of a test, and the interrelationship between 
the^two are made explicit. 



ERIC 



D. ACCURACY STANDARDS 



11. Obi ective Reporting . The evaluation procedures should provide 

safeguards to protect the evaluatidfn findings and reports against 
distortion by the personal feelings and biases of any party to the- 
evaluation. ' ^ 



Note: See also the references for Standards A2, A4, B2,^ C2, C3, C7, D2, 
D4, D5, D6, and DlO. , 



Datta Lois-ellen. Communicating Evaluation Results for Policy Decision 

Making. In Berk,. Ronald A. (Ed.), Educatiorial Evaluation Methodo lo^^; 
The State of the Art . Baltimore: Johns Hopkins Univ. Press, 
Chapter 6. 

This chapter reviews techniques and issues in communicating evaluation 
findings to decision-makers. The lit4rature includes studies of techniques 
sich as adversary evaluation, reports of stakeholder participation studies, 
and case studies of " evaluation, utilization. Actual evaluation «P°^ts from 
ten large city school districts and five states are critiqued; the mrfjority 
are characterized as descriptive statistical accounts rather than action- . 
oriented Evaluations. It is argued that effective communication must come 
at the beginning of an evaluation study as -an integral part of the planning 
and execution. 



GENERAL MONOGRAPHS AND TEXTBOOKS 



Anderson. Scarvia B.; Ball, Samuel. The Profession and Practice of 
Program Evaluation . San Francisco: Jossey-Bass, 197S. 

Beeinning with an overview of the field. this\.ook treats the following 
toScs the major purposes of evaluating educational and social programs; 
general- method 'of evaluation best suited for each purpose; typ.s and 
Sources of evidence frequently associated with the general methods of 
investigation; targeted dissemination of evaluation information and 
ielul^s- tie professional predispositions and preferences of evaluators 
that ma;- influence what th.y look at and how they look at it; the complex 
fiscal anJ adhinistrative relationships among funding agencies, program 
direc o". anS evaluators; ethical responsibilities bound up in program 
evaluation; defining, instilling, and assessing the competencies of - 
eva ua ors an.l the status and prospects for evaluation as a "profession. 



Apple, Michael W.; And Others (Eds.). Educational f ^"g^f 
Lnd Responsibility . Berkeley. Calif.: McCutchan Pub. uo.. iW4. 

This volume i« concerned with three problems of major significance. 
This volume ir . .^ing the perspectives of educators on the problems. 
of"and appri: h^ to'va^iafion? going beyond the restrictive input-output 
: L U :rat d;:mtnate the topic. Secondly it is aimed - overcoming the 
limited range of conceptual and historical '^^%^73^fi^°" 
field Final.' y, it portrays the intense controvefty that any serious 
d :cu;siL of e;alua?ion is bound to raise After -^"^^^-^^J^^.i^^ 
chapters, six chapters on the concepts, values-and methods m evaluation 
are each follcved by a critique and discussion. 



Babbie. Earl R. Survey Research Methods . Belmont , Calif . : Wadsworth Pub. , 
1973. 

This book is addressed to three problems related to the misconception that 
.lirvey research is simple. First, the faddish popularity °| ^-ey^jf .^'^^ 
has inevitably resulted in a large number of bad surveys. Second, the 
lileZread overu.e and misuse of survey methods has led to ^he wholesale 
"election of survey research by many people. Third, the f^J^/ 
Sen survey was poorly conducted incorrectly presupposes an established 
body of Bcientrfir.tandards againsf which to evaluate survey fctiv.ties 
T^is ?ext focuses on the logic and skills of- survey research, in an attempt 

provide a practical guide to survey research for students and other 
prospective researchers. ; 
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Baker. Eva L.^ Quellmalz. Edys S. (Edsff Educational Testing and 
. Design. Analysis, and Policy . . Bev^ly Hills. Calit.: SAGE Pub.. 1980. 

The papers collected here were presented at an invitational conference on 
Measurement and Methodology in Education, sponsored ' 
Study of Evaluation. In general, the participants paid relat.ively UtC-le 
attention to educational policy as a legitimate focus for activity, nor . 
do they concretely express concern for students, those who are the data 
providers in most testing and evaluation efforts. They do directly 
address issues iti test design, quantitative theory and applications, and 
evaluation and testing policy. 

Banner, David K.; And Others, The'Politics of Social Program Evaluation. 
Cambridge. Mass.: Bellinger Pub. Co., l975. 

Attempts to develop evaluation procedures for social programs have been • ^ 
frauRht with difficulties, obstacles, and political barriers. These are - 
descfiJed in.this book in* the context of the early history OPP°^ 
tunity Funding Corporation (OFG), a demonstration activity establishetf- 
and funded by the Office of Economic Opportunity during-the first Ni^gn 
:SminSration. This case st^dy relies primarily on P-^-^ ^ "-^^ 
and selected secondary sources; the data were then analyzed in light of 
relevant theory on the politics of evaluation. 

Bennett. Carl A.; Lumsdaine. Arthur A. (Eds.) ' Evaluation and Experiment : 
Some Critical Issues in Ass essing Social Programs. New York. 
Academic Press, 1975. . • 

This volume is an outgrowth of a symposium held at the Battelle Seattle 
Research Center in July. 1973, It focuses on some selected aspects ot 
the p oblems in evaluating the outcomes of socially important Programs. 
Its eight, chapters cover the defining of evaluation issues; assessment as 
arempiHcal base for policy; effect size estimation in quasi-experimental 
design's! regression and selection models to improve nonexperimental com- 
pari8on^. field trial designs in gauging the impact of fertility planning • 
llogrZl a reexamination of experiments and evaluations; 0P«"t^°?;l 
systematic research on production, maintenance, control, and adaptive 
functions of feedback; .and assessing alternative conceptions of evaluation^^ 
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Borlch, Gary D, (Ed.) Evaluating 'Educational Programs and Products , 
Englewood Cliffs, TOTi Educational Technology fub., 

This book is a* guide and handbook for planners, developers, and evaluators 
of educational programs and pr<^ucts. Its purpose is to provide P^^^icai 
insights that- are immediately a'^plicable to planning and executing effec 
tive progra^ and produtt evaluations. It divides the evaluator s work ■ 
into three activities: establishing perspective, or choosing an appropriate 
role fo^ the context" ip^hich he or she will work; planning the evaluation, . 
or choosing an appropriate model or strategy; and analyzing the data or 
selecting appropriate^ analysis" methods and techniques. The task of this 
book is to identify specific procedures that are appropriate to each of 
these . act ivi ties , 

* * 

f 

Campbell, Donald T,; Stanley, Julian C Experimental and ^^^7 
. Experimental Designs for Research . Chicago: Rand McNally & Co., 1963. ^ 

the validity of 16 experimental designs against ,12 commoa threats to valid 
inference is examinedC The designs examined are the one-shot case study, ^ 
the one-group pretest-posttest design, the static-group comparison, the 
pretest-posttest control .group design, the Solomon four-group design, the 
posttest-only control group design, the time-series experiment, the 

^uivalent time-samples design, the «<^"i^«l«"^"•^'^""^,^^""«": '3" a^ate- 
nonequivalent control group design, counter balanced designs, the separate 
sample pretest-posttegt design, the , separate-sample pretest-posttest ^ 
control group design, the multiple time-series '^^"g"' ^ V'^" 
institutional cycle design, and r-egtession-dxscontinuity dhalyais.^ 

Caro, Francis G. (Ed.) Readings in E valuation Research. Second .Edition. 
New York: Russell Sage Foundation, 1977. 

This volume brings together material about evaluation research awn from 
?iariety of sources, an^ includes both general statements about evaluation 
research and specific case materials. The general articles address such 
Csues as the nature of the evaluation task, the role of evaluation research 
in programs of directed change, tl* organizational context in which evaluation 
•research is conducted, and the methodological strategies appropriate for | 
evaluation research. The case materials include^treatment of P^^^/^J j 
in the establishment, of the evaluation «search role, and reports of findirfgs 
of completed evaluation research -studies^ ^ 
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Cook, Thomas D.; And Others (Eds.) Evaluation S tudies Review Annual. 
Volume 3 . Beverly Hills, Calif.: SAGE Pub., iy/8. 

The third volume in this series includes papers on the policy and political 
context of evaluation, raeihodology, and exemplary studies from the fields 
of health, income maintenance, criminal justice, education, mental health,- 
and evaluations in the "public interest." 



Cook, Thomas D.; ReichardC, Charles S. (Eds.") Qualitative and Quanti- 
tative Methods in Evaluation Research . Fever ly Hills, Calit.: bAbt, 
• Pub., 1979. 

A diversity of opinion on the use of qualitative and quantitative methods 
was purposely included in this volume. William J. Filstead distinguishes 
between the qualitative and quantitative paradigms, and argues that the 
qualitative paridigm is most appropriate for evaluation research. Donald 
T. Campbell provides a convincing rationale for the use of qualitative 
methods in the case study design. M. G. Trend describes an evaluation of 
a maior housing allowance demonstration which used both participant 
observers and questionnaire surveys. Francis A. J. lanni and Margaret 
Terry Orr argue that evaluat6rs can profitably use ethnographic techniques 
only if these methods are carefully adapted to fit the neecis of evaluation 
research. Howard S. Becker discusses the "truth" of photographic evidence. 
Michael S. Knapp provides an examination of the us£ of ethnography In the 
evaluation of the Experimental Schools Program." Robinson G. Hollister, 
Peter Kemper, and Judith Wooldridge offer some insighfs into the use ot 
quantitative methods. 



Cronliach, Lee J.; And Others. Toward Reform of -Program Evaluation. 
San Francisco: Jossey-Bass, 1980. 

In an attempt to bring about change in thought by provoking argument, the 
main ideas ot the book are first presented as, 95 theses. Th.se theses 
include statements about the role of evaluation; the political impact of 
an evaluation; the usefulness of evaluation for decisionmaking; the , 
information dissemination role of evaluation; the relation between^ 
program goals and evaluation; the design of an evaluation; evaluation 
validity; evaluation as a profession; and the various roles of the 
evaluator. 
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•Cronbach, Lee* J. Essentials of PavcholoRical Testing . Third Edition. 
New York: Harper & Row, 1970. 

r 

This book' is inten<led to establish a base of .the essentials of measurement: 
methods of inquiry, the critical standards, and the key concepts of the 
TJIT °L .LJ.r. address the purposes and types of tests^ test adminis- 

ICS desired \n tests; 
Lty measures'. 



Datta. Lois-ellin; Perloff. Robert (Eds.) Improving Evaluations. 
Beverly Hills, Calif.: SAGE Pub., 1979. 

Dedicated to Marcia Guttentag, the four sections of this 

interests prominent in her. published work. .The first ««f f 

the political forces influencing what questions are asked, when, and by 

whom. Writers in the second section share a common ^^""^^"^J^^^^o^^bly 

in which it is' inextricably embedded. 

c 

Freeman. Howard E.; Solomon, Marian A. (Eds.) Evaluation Studies 

Review Annual. Volume 6 . Beverly Kills, Calif . : SAGE Pub.. .1981. 

The sixth volume in this series indues papers °?^/°""P'=%^"?,f J^Suca'^ 
and evaluation utilization, and exemplary studi^ from the fields °f 
lion, human resources and social services, law and public safety, health, 
•mental health and substance use, and environment.. 

Glass, Gelle V. (Ed.) Evaluation Studies Rev iew Annual. Volume 1. 
yeverly Hills, Calif..: SAGE Pub.. 1976. 

The^irst volume in this series includes papers on the theory and methods 
of evaluation, and exemplary studies from the fields of education, men al 
health and public health services, welfare and social services, and crime 
and justice. 
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Glass. Gene V.; Stanley, Julian C. Methods in Educaclon an^ 

Psychology . Englewood Cliffs, N.J. : Prentxce-Hall, 1970. 

This textbook is for use in statistics courses in education and the 
social sciences. Its chapters cover measurement, scales, and statistics; 
tabulating and depicting data; measures o^ central tendency; measures of 
iariabilUy; the normal distribution; mea'sures of r^ationship; prediction 
and estimation; probability; statistical inference; analysis of variances, 
multiple compa;ison procedures; and fundamentals of experimental design. 

Grotelueschen, Arden D. ; And Others. An Evaluation Planner; A Guidebook 
for nPVPlo ping Evaluation Plans Responsive to a Variety of Contexts., 
■ Anrj fen ces, and Issues Withi n Adult Basi c Education . U^ana: Univ. 
of Illinois, Office^or the Study ot continuing Professional 
I Education, 1974. 

Systematic guidelines for evaluating programs in adult basic education 
a^e pr^ ent!d. Program activities in adult basic education J^volve four 
K.ain levels of activity (classroom, Ipcal, state, and federal), each of 
which cin profit from appropriately focused evaluation. Patterns of 
assoaatLS aming persoSSel and activities are present in each educational 
conteir as well as ponrmon evaluation needs and emphasis. Typical charac- . 
teristics of each context are described. Eight evaluation components 
f"m tie basis of the planner: purposes, audiences, issues. 
evidence, data-gathering, analysis, and reporting. A discussion of each 
loZnenl is foUowed by a checklist containing 

and techniques in developing an evaluation design for each of the tour . 
contexts. 
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Guba, Egon G.; Lincoln, Yvonna S. Effective Eva luation: Itnprov/ng .the 
Usefulness of E^^luation Resu lts Through Responsive, and Naturalistic 
Approaches . San Francisco: Jossey-Bass, 1981. 

This book pteWts a neW model of evaluation-one that organizes evaluation 
•activities so that it illuminates the claims, concerns and issues raised _ 
by stakeholding audiences (responsive evaluation) and ."^'^""^"'^"^ 
methodologies to gather information-. A variety of evaluation models are 
analyzed to provide background information necessary to an understanding of • 
"e responsive approach. Scientific and naturalistic inquiry paradigms are 
coLareS and a c«se is mude that the latter be used in the study of hunjan , 
behavior: Problems associated with-u«ing a human being as an assessment 
instrument are discussed as well as ways of improving the human ^nstrume^t 
i^^rp isalso a sferies of chapters discussing the methods and methodologies 
I^fociated w?th quaitaJlve inquiry. The last part of the book is a present- 
ation of steps bj which naturalistic-responsive evaluat on is carried out 
tlnl the item^ discussed are contracting for an evaluation, establishing the 
e^aJuator/presence on the site, developing contacts, avoiding ^-rinvolve- 
^Int and cooptation and dealing with human and ppUtical problems. Th"^J^« 

discussion on how to deal with audiences and elicit their claims concerns 
;irissues ?Se methods of identifying the Kinds and sources of useful 
info matio; are presented. The book concludes with a chapter on^^P-^^^' 
emphasizing different audiences, different reporting requirements, and 
that reporting is coni:inuous. 

Guttentag, Marcia; Saar. Shalom (Eds.) Evaluation Studies Review Annual, 
Volume 2 . Beverly Hills, Calif . : SAGE Pub.. 1977. 

The second volume in this series includes papers on "^^if ^Jf „f 
ation, evaluation methodology and data integration, evaluation into policy, 
and exemplary studies from the fields of education, crime and justice, 
and human services. 

GuTtentag. Marcia; Struening. Elmer L. (Eds.) Handbook of Evaluation 
• Research . Two volumes. Beverly Hills, Calif . : SAGE Pub... 1975. 

One, purpose of this Handbook is to provide evaluators with a sample of 
experts! a panel of consultants between hard covers, with whom they can 
colunicate as they develojx.the crucial steps of their studies. The 
Handbook is also designed to serve as a textbook for courses m ^^aluat^n 
at the graduate level. Ihe first volume ^TPhasi^essei^tegies and methods 
of evaluation, while the second volume reviews the literature in selected 
content areas. 
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Hamilton, David; And Others (Eds.) Beyond the Numbers Gam e: A' Reader in 
Educational Evaluation . London: Macmil Ian- Education, 197?. 

This book charts the paradigm shift from an evaluation methodology valuing 
numeracy to one valuing literacy. The many readings are organized into 
four sections: "The Objectives Model Revisited," "Five Advocates of 
Change" (Myron Atkin. Elliot Eisner, Lawrence Stenhouse, Michael Scnven, 
and Robert Stake), "Alternative Methodology," and "Alternative Evaluation: 
The New Paradigm in Action." 



Hays, William L. Statistics for Psychologists . Second Edition. New 
York: Holt, Rinehart and Winston, 1973. 

This book represents an attempt to give the elements of modern statistics 
in a relatively nonmathematical form, but in somewhat more detail than is 
customary in texts designed for psychologists, and with* considerably more 
emphasis on the theoretical rather than the applied. aspects of the 8ubj.ect, 
Its chapters address sets and functions, elementary" probability theory, 
the binomial distribution, central tendency and variability, sampling 
distributions and point estimation, the normal distribut;ioA, hypothesis 
testing, inferences about population means, the chi-square and F d\stri- 
butions, analysis of variance, and linear regression and correlation. - 



Heise, Pavid R. Causa?. Analysis . New York: John Wiley & Sons, 1975. 

This book focuses on the study of linear systems and ref)resents an effort 
to organize a broad range of information' about this topic m a fairly 
elementary fashion. Its chapters consider causality and causal analysis, 
causal diagrams and flowgraph analysis, statistical concepts, path 
analysis, identification and estimation, and dynamic considerations. 



House, Ernest R. (Ed.) School Evaluation: The P olitics & Process. 
Berkeley, Calif.: McCutchan Pub. Co., 1973. 

The primary purpose of this book is to acquaint the practicing administra- ^ 
tor, the decision-maker, and the educational-consumer with the world of 
evaluation. Its mljor theme is the political nature of evaluation. A 
second theme is that both^ne's administration and one's evaluation are ^ 
intimately related to whether one believes that the goals of the class, 
school district, state, or nation are already established or are yet to 
be arrrived at through negotiation between groups. A third theme is the 
relationship between decision-making and ^evaluation: the delicate relation 
ship between the Administrator and the eValuator. 
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Morris, Lynn Lyons; And Others. Program Ev&lu ation Kit. Beverly Hills, 
Calif.: SAGE Pub., 1978. 5 ^ 

The Kit is a set of books intended to assist people who are conducting 
evaluations of educational programs. It consists of the ^--SJ^f^^ 
books: ThP Evaluator-s Hartdbook ; H ow to Deal With Goals and Ob,i^ctxves 
How to D esign a Pcogra ^TE^JIuIiFronT How to Measure Program Implementation . 
■ h°" to Measure Attitudes ; »»» rolleasurc Achievement ; How to CaLcu-U^ 
g^n^^c^jrBr and How to Presjtnt an E valuation Report. 



Patton. Michael Quinn. Qualitative Ev aluation Methods. Beverly Hills, 
Calif.: SAGE Pub., 19ciO. 

This book is designed to explain in which evaluation situations >alitativ^ 
methods are useful, and how to actuall;y Ose those methods. It is organized 
Tround tSree topic^: conceptual issued in the use of <l"f "^^^ ^f.f ^ 
for evaluation research, collecting qualitative data, and data analysis. 



Popham, W. James. Educational Evaluation . Englewood Cliffs, 
Prentice-Hall. 1975. 



N.J. : 



This textbook is designed to be appropriate for beginning 
evaluators. and graduate students in educational evaluation. Its chapters 
addrefs co;tempo?ary^conceptions of evaluation, instructional objectives, 
the use of measurement, expanding measurement alternatives, classical 
Measurement considerations, criterion-referenced measurement. -asu,ement 
of affect, evaluation designs, sampli.,g strategies. analyi:ing evaluative 
data, reporting evaluation results, cost analysis considerations, and 
teacher evaluation. 



Popham. W. James (Ed.) Evaluation in Education: 
Berkeley. Calif.: McCutchan Pub. Go.. 1974. 



Current Applications. 



This volume was initiated by the Standing -Committee on Research Training 
'>?f hl A^erican Educational Research Association ^^JP^!^1^^;: ' 

"Evaluation Perspectives and Procedures." by Michael Scriven, Alternative 
A^^riacJes to Educational Evaluation: A ^elf-Study Guide for Eva uatrs. 
by David L. Stufflebeam; "Designing Summative Evaluation Studiep the 
tIcbI Level." by Peter W. Airasian; "Data Analysis and Reporting Consider 
^iois in Evaluation." by Richard M. Wolf; "The Use of Standardized Tests 
in -E^ li^tion." by Gilbert Sax; "Criterion-Referenced Measurement "by Jason 
MiUman; "Cost Analysis for Educational Program Evaluation.' by ^mil J- 
Ha ler; "Introduction to Matrix Sampling for the V^J^^^^^^^'*^ 
A. Siritnik; and "Formative Evaluation of Insti^uction." by Eva L. Baker. 
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Rosenthal, Robert; Rosnow, Ralph L, (Eds.)' Artifact in Behavioral 
. Research . New York: Academic Presfe, 1969. 

That portion of the complexity of human behavior which can be a-tributed 
toTthe social nature of behavioral research can be conceptualized as a 
set of artifacts to be ' olated, measured, considered and, sometimes, 
eliminated. This book i« desig' ^d to consider in detail a number of 
these artifacts. .The introductory chapter provides a perspective on 
artifact and a discussion of the nature of experimental control. The 
following six chapters are a series of position papers by researchers who 
have been actively engaged in systematic exploration of various ante- 
cedents of artifact iti behavioral research, and each writer summarizes 
the findings in his respective area: suspiciousness of intent, volunteer 
effects, pretest sensitization, demand chAracteristics. experimenter 
expectancy effects, and evaluation apprehension. The final chapter takes 
!hto account the sep rate contributions' and discusses the future prospects 
for behavioral research. 



Rossi, Peter H. ; Williams, Walter (Eds.) Evaluating Social PEOgramsi 
Theory. Practice . -and Politics . New York:. Seminar frees, 1972. 

This volume of readings is divided into four sections : An Overview; 
Evaluative Research: Theory; Evaluative Resea^h: " 
nizing for Large-Scale Evaluative Research. The papers in the- Theory 
section address statistical design requirements and methodological^ 
issued. The papers in the Practice section illustrate evaluation in 
education, Aoor programs, and income maintenance experiments. The final 
section discusses the r^jles in evaluation research of various social 
institutions. 



Sechrest, Lee; And Others (Eds.) Evaluation Studies Review Annual. 
Volume 4 . Beverly Hills, Calif . : SAGE Pub. , 1979. 

The fourth volume in this series includes papers on the theory and 
philosophy of evaluation, alternative methodologies and strategies, 
the technology of evaluation, unanticipated findings, evaluation utili- 
zation, and a variety of evaluation studies. 



I". 



-103- 



Sechrest, Lee (Ed.) Unobtrusive Measurement" Today. New Directions for 
Methodology of Behavioral Science , nl, 1979. 

The focus of this volume is on the' appropriatfe utilization of unobtrusive 
measures in research. An overview addresses some of the critical methodolo 
gicaiand conceptual issues that faces the researcher wishing to devise and 
Lploy an unobtrusive measure. Other chapters include discussions on direct 
observation, application in field experiments «nd cross-cultural research, 
honverbal behaviors, and tKe physical trace. 



Stromsdorfer, Ernst W. ; Farkas, George (Eds ) ^^^j^f Studies Review 
Annual. Vtflume 5 . Beverly Hills, Calif.: SAGE Pub., iViiU. 

• « 

The fifth volume in this series includes papers on methodology, and exemplary 
re:ig;erand natural experiments from the fields of l^^or. education housing, 
health and safety, energy and resources, and public financial policy. 



Thorndike, Robert L. ; Hagen, Elizabeth P. Measurement and E^^^^^^;"" ^" 
Psychology and Education . Fourth Edition. New York: Jonn wiley & 
Sons, 1977. 

The chapters of this measurement textbook address: measurement and numbers; 
qualities desired in a measurement procedure; norms; various types of tests 
and measurement instruments; planning a school testing program; grading; and 
social and political issues in testing. 



Tuckman, Bruce W. Conducting Educational Research . New York: Harcourt 
Brace Jovanovich, 1972. 

\ 

This book is based on the premises that research is a useful tool for 
educators, and that much educational research,must be undertaken xn the 
field- It discusses the role of research, selecting a problem and con- 
structing hypothesis, identifying and labeling variables, constructing 
operational definitions of varia1,les, identifying techniques for the 
manipulation and control of variables, and constructing research designs 
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Tyler, Ralph W.; And Others (Eds.) Perspectives of Curriculum Evaluation.. 
Chicago: Rand McNally & Co., 1967. 

This volume is the first in a Monograph Series on Curriculum Evaluation, 
developed by the American Educational Research Association. Its chapters 
include: "Toward a Technology for the Evaluation of Educational Programs, 
by Robert E. Stake; "Changing Concepts of Educational Evaluation, by 
RalDh W. Tyler; "Curriculum Research and the Promotion of Learning, ■J>y 
Robert M.Gagn;; "The Methodology of Evaluation "by Michae Scriven; and 
"Aspects of Curriculum Evaluation: A Synopsis," by J. Stanley Ahmann. 

I 

Weiss, Carol H. Evaluating Action Programs: Readings in Social Action and 
Education , Boston: Allyn and Bacon, iV/il. 

'* 

This book aims to help the trader conceptualize and understand the purposes 
of evaluation and the methods by which it obtains information and 8^"^"^^= ^ 
conclusions. ' Rather than giving a set of prefabricated rules ^"'^/"f,^^"^^^^^"^* 
U points out the constraints within which evaluation operates and suggests 
alternative strategies of design, measurement, structure, relation hip and 
communication in order to accommodate to existing constraints and to serve 
the informational needs of programs. 

Weiss, Carol H. Evaluation Res — y^^r'y.r.A<^ for Assessing Program 
Effectiveness . Englewood Cliffs , N.J. : Prentice-Hall, 1972. 

This book deals with the application of research methods to the evaluation - 
of social programs: programs in education, social work, corrections, 
health, mental health, job training, technical assistance, community ^ 
action and law. Its basic theme is that evaluation uses the method s^ifd 
tools of social research but applies them in an action context that . 
intrinsically inhospitable to them. Its chapters discuss the purposes 6f 
evaluation, formulating the question and measuring the answer, design of 
the evaluation, the turbulent setting of the action program, and utili 
zation of evaluation results. 

Worthen. Blaine R. ; Sanders, James R. Educational . '^^?;y3 

and Practice . Worthington, Ohio: Charles A. JOnes fub. w., i:»/3. 

This book pulls together in one volume the best of the emerging literature 
on educational evaluation, and identifies and fills serious g^ps in the 
Uterature. It is organized ground four topics: evaluation as disciplined 
wlr^frameworks for planning evaluatior; studies, considerations m 
planning evaluation studies, and the future^^of evaluation. 
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